通知

すべてクリア

slurm problem

最新の投稿

RSS

台坂博

(@daisaka)

Estimable Member

結合: 9年前

投稿: 116

Topic starter 06/05/2019 4:44 pm

一橋大学の台坂です。いつもお世話になっております。

slurm の grand, mem64, mem32のpartitionがdownしております。復旧して頂けると助かります。

また、t001k09n02 も不調のようです。こちらも見て頂けると助かります。

引用

Shinichi Hirahara

(@hira)

Estimable Member Admin

結合: 8年前

投稿: 106

08/05/2019 4:53 pm

台坂先生

エクサ平原です｡

お世話になります｡

対応遅くなり申し訳ありません｡

ご指摘のPartitonに絡むノードを復旧しました｡

現在すべてアクティブです｡

t001k09もブリックごと再起動済みです｡

宜しくお願いいたします｡

返信引用

台坂博

(@daisaka)

Estimable Member

結合: 9年前

投稿: 116

Topic starter 08/05/2019 7:14 pm

平原様、

一橋大学のいつもお世話になっております。

sinfo を実行すると、以前、mem32, mem64, grand がダウンしているようです。

再度、確認をお願いできるでしょうか？

お手数をおかけしますが、よろしくお願いいたします。

返信引用

Shinichi Hirahara

(@hira)

Estimable Member Admin

結合: 8年前

投稿: 106

09/05/2019 9:23 am

台坂先生

失礼しました

現在すべてのPartitonがupになっています｡

宜しくお願いします｡

返信引用

台坂博

(@daisaka)

Estimable Member

結合: 9年前

投稿: 116

Topic starter 09/05/2019 1:27 pm

平原様、

一橋大学のいつもお世話になっております。

up になっていることを確認しました。どうもありがとうございました。

返信引用

台坂博

(@daisaka)

Estimable Member

結合: 9年前

投稿: 116

Topic starter 09/05/2019 1:38 pm

平原様

一橋大学の台坂です。度々すみません。

t001k09n02 のSC2のいずれかのボードが不調のようです。計算がまた止まってしまいました。

調整をお願いいたします。

返信引用

Shinichi Hirahara

(@hira)

Estimable Member Admin

結合: 8年前

投稿: 106

09/05/2019 5:46 pm

台坂先生

対応が遅くなり申し訳ありません｡

t001k09ですが､先ほど調整を終了し､こちら側のテストをパスしましたので

SlurmをDrainからIdleへ戻しております｡

宜しくお願いいたします｡

返信引用

台坂博

(@daisaka)

Estimable Member

結合: 9年前

投稿: 116

Topic starter 09/05/2019 6:41 pm

平原様

一橋大学の台坂です。どうもありがとうございます。

mem64 のパーティションで、MPIがうまく起動できないようです。

slurm scriptで指定しているnode list 先頭のノードで、以下のようになっているようです。

daisaka@t001k02n01's password:
daisaka 76280 97.8 0.0 431528 15492 ? Rl 17:22 40:51 mpirun ./x47_v2_Q_fixtest_allinput_pzoclmpi 200 1 200 200 200 200 200 200 1.15 -82 1.2 0.0 1.0 0.0 1.0 0.0 1.0 0.0 1.0 0.0 1.0 0.0 1.0 0.0 1.0 3.75 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0
daisaka 76372 0.0 0.0 0 0 ? Z 17:23 0:00 [x47_v2_Q_fixtes] <defunct>
daisaka 76375 0.0 0.0 0 0 ? Z 17:23 0:00 [x47_v2_Q_fixtes] <defunct>
daisaka 76376 0.0 0.0 0 0 ? Z 17:23 0:00 [x47_v2_Q_fixtes] <defunct>
daisaka 76377 0.0 0.0 0 0 ? Z 17:23 0:00 [x47_v2_Q_fixtes] <defunct>
daisaka 76378 0.0 0.0 0 0 ? Z 17:23 0:00 [x47_v2_Q_fixtes] <defunct>
daisaka 76379 0.0 0.0 0 0 ? Z 17:23 0:00 [x47_v2_Q_fixtes] <defunct>
daisaka 76380 0.0 0.0 0 0 ? Z 17:23 0:00 [x47_v2_Q_fixtes] <defunct>
daisaka 76381 0.0 0.0 0 0 ? Z 17:23 0:00 [x47_v2_Q_fixtes] <defunct>

お手数ですが、確認をお願いできると助かります。

返信引用

台坂博

(@daisaka)

Estimable Member

結合: 9年前

投稿: 116

Topic starter 09/05/2019 7:27 pm

追伸ですが、batch でも走らなくなってしまいました。

slurm log に、

A process or daemon was unable to complete a TCP connection
to another process:
Local host: t001k01n01
Remote host: t001k09n01
This is usually caused by a firewall on the remote host. Please
check that any firewall (e.g., iptables) has been disabled and
try again.
------------------------------------------------------------
[t001k02n01:76855] Error: pml_yalla.c:97 - recv_ep_address() Failed to receive EP address

等々、エラーがでていました。

また、es1fe のレスポンスがちょっと悪い感じがあります。

何か、ネットワークワークの問題はありませんか？

返信引用

Shinichi Hirahara

(@hira)

Estimable Member Admin

結合: 8年前

投稿: 106

09/05/2019 8:14 pm

台坂先生

本日15時くらいからes1feのレスポンスが遅くなっていることは認識しており､現在調査してもらっています｡

宜しくお願いいたします｡

返信引用

yamaura

(@yamaura)

Eminent Member Admin

結合: 8年前

投稿: 20

09/05/2019 10:55 pm

レスポンスが遅いのはネットワークではないのですが、原因が分からなく、埒が明かないので、
今利用者もいないようなので、明朝F/Eマシンの再起動を試みようかと思います。

返信引用

Shinichi Hirahara

(@hira)

Estimable Member Admin

結合: 8年前

投稿: 106

10/05/2019 11:08 am

台坂先生

エクサ平原です｡

フロントエンドの再起動でレスポンスの遅延は復旧いたしました｡

mem64､mem32でのMPIの動作も確認しております｡

宜しくお願いいたします｡

返信引用

台坂博

(@daisaka)

Estimable Member

結合: 9年前

投稿: 116

Topic starter 10/05/2019 2:19 pm

平原様、

一橋大学の台坂です。対応をどうもありがとうございます。

jobは走るようになりました。es1feのレスポンスも良くなりました。

お手数をおかけしました。

返信引用

Super Globals

Requests: Array ( )

Server: Array ( )

Options and Features

permastruct: community
use_home_url: 0
url: https://portal.pezy.jp/community/
Array
(
    [title] => PEZY User Portal フォーラム
    [description] => PEZY User Portal ディスカッション掲示板
    [lang] => 1
    [menu_position] => 23
)
pageid:100
default_groupid: 3
Array
(
    [layout_qa_intro_topics_toggle] => 1
    [layout_extended_intro_topics_toggle] => 1
    [layout_qa_intro_topics_count] => 3
    [layout_extended_intro_topics_count] => 5
    [layout_qa_intro_topics_length] => 90
    [layout_extended_intro_topics_length] => 45
    [display_current_viewers] => 1
    [layout_threaded_intro_topics_toggle] => 0
    [layout_threaded_display_subforums] => 1
    [layout_threaded_intro_topics_count] => 10
    [layout_threaded_intro_topics_length] => 0
    [layout_threaded_filter_buttons] => 1
    [layout_threaded_add_topic_button] => 1
)
Array
(
    [layout_extended_intro_posts_toggle] => 1
    [layout_extended_intro_posts_count] => 4
    [layout_extended_intro_posts_length] => 50
    [recent_posts_type] => topics
    [tags] => 1
    [max_tags] => 5
    [tags_per_page] => 100
    [topics_per_page] => 10
    [edit_topic] => 1
    [edit_post] => 1
    [eot_durr] => 300
    [dot_durr] => 300
    [posts_per_page] => 15
    [eor_durr] => 300
    [dor_durr] => 300
    [max_upload_size] => 52428800
    [display_current_viewers] => 1
    [display_recent_viewers] => 1
    [display_admin_viewers] => 1
    [attach_cant_view_msg] => この添付ファイルの表示は許可されていません
    [layout_threaded_posts_per_page] => 5
    [layout_qa_posts_per_page] => 15
    [layout_qa_comments_limit_count] => 3
    [layout_qa_first_post_reply] => 1
    [layout_threaded_nesting_level] => 5
    [layout_threaded_first_post_reply] => 0
    [union_first_post] => Array
        (
            [1] => 0
            [2] => 0
            [3] => 1
            [4] => 0
        )

[search_max_results] => 100
    [topic_body_min_length] => 2
    [topic_body_max_length] => 0
    [post_body_min_length] => 2
    [post_body_max_length] => 0
    [comment_body_min_length] => 2
    [comment_body_max_length] => 0
    [toolbar_location_topic] => top
    [toolbar_location_reply] => top
)
Array
(
    [custom_title_is_on] => 1
    [default_title] => Member
    [members_per_page] => 15
    [online_status_timeout] => 240
    [url_structure] => nicename
    [search_type] => search
    [login_url] => 
    [register_url] => 
    [lost_password_url] => 
    [redirect_url_after_login] => 
    [redirect_url_after_register] => 
    [redirect_url_after_confirm_sbscrb] => 
    [title_usergroup] => Array
        (
            [1] => 1
            [5] => 1
            [4] => 1
            [2] => 1
            [3] => 0
        )

[title_second_usergroup] => Array
        (
            [1] => 0
            [5] => 0
            [4] => 0
            [2] => 0
            [3] => 1
        )

[rating_title_ug] => Array
        (
            [1] => 1
            [5] => 1
            [4] => 1
            [2] => 1
            [3] => 1
        )

[rating_badge_ug] => Array
        (
            [1] => 1
            [5] => 1
            [4] => 1
            [2] => 1
            [3] => 1
        )

)
Array
(
 [from_name] => PEZY User Portal - Forum
 [from_email] => pz-user-internal@pezy.co.jp
 [admin_emails] => pz-user-internal@pezy.co.jp
 [new_topic_notify] => 1
 [new_reply_notify] => 1
 [confirmation_email_subject] => Please confirm subscription to [entry_title]
 [confirmation_email_message] => Hello [member_name]! 
 Thank you for subscribing. 
 This is an automated response. 
 We are glad to inform you that after confirmation you will get updates from - [entry_title]. 
 Please click on link below to complete this step. 
 [confirm_link]
 [new_topic_notification_email_subject] => New Topic
 [new_topic_notification_email_message] => Hello [member_name]! 
 New topic has been created on your subscribed forum - [forum].
 
 [topic_title]
 <blockquote>
 [topic_desc]
 </blockquote>
 <hr>
 If you want to unsubscribe from this forum please use the link below. 
 [unsubscribe_link]
 [new_post_notification_email_subject] => New Reply
 [new_post_notification_email_message] => Hello [member_name]! 
 New reply has been posted on your subscribed topic - [topic].
 
 [reply_title]
 <blockquote >
 [reply_desc]
 </blockquote>
 <hr>
 If you want to unsubscribe from this topic please use the link below. 
 [unsubscribe_link]
 [report_email_subject] => Forum Post Report
 [report_email_message] => Report details:
 Reporter: [reporter], 
 Message: [message], 
 
 [post_url]
 [wp_new_user_notification_email_admin_subject] => [blogname] New User Registration
 [wp_new_user_notification_email_admin_message] => New user registration on your site [blogname]:

Username: [user_login]

Email: [user_email]

[wp_new_user_notification_email_subject] => [blogname] Your username and password info
    [wp_new_user_notification_email_message] => Username: [user_login]

To set your password, visit the following address:

[set_password_url]

[reset_password_email_message] => Hello!

You asked us to reset your password for your account using the email address [user_login].

If this was a mistake, or you didn\'t ask for a password reset, just ignore this email and nothing will happen.

To reset your password, visit the following address:

[reset_password_url]

Thanks!
 [update] => 1
 [user_mention_notify] => 1
 [user_mention_email_subject] => You have been mentioned in forum post
 [user_mention_email_message] => Hi [mentioned-user-name]! 
 You have been mentioned in a post on \"[topic-title]\" by [author-user-name].

Post URL: [post-url]
    [overwrite_new_user_notification_admin] => 1
    [overwrite_new_user_notification] => 1
    [overwrite_reset_password_email_message] => 1
)
Array
(
    [user-admin-bar] => 0
    [page-title] => 1
    [top-bar] => 0
    [top-bar-search] => 0
    [breadcrumb] => 1
    [footer-stat] => 0
    [mention-nicknames] => 1
    [content-do_shortcode] => 0
    [view-logging] => 1
    [track-logging] => 1
    [author-link] => 0
    [comment-author-link] => 0
    [user-register] => 0
    [user-register-email-confirm] => 0
    [register-url] => 0
    [login-url] => 0
    [resetpass-url] => 0
    [replace-avatar] => 1
    [avatars] => 1
    [custom-avatars] => 1
    [signature] => 1
    [rating] => 1
    [rating_title] => 1
    [member_cashe] => 1
    [object_cashe] => 1
    [html_cashe] => 0
    [memory_cashe] => 1
    [seo-title] => 1
    [seo-meta] => 1
    [seo-profile] => 1
    [rss-feed] => 1
    [font-awesome] => 1
    [bp_profile] => 0
    [bp_activity] => 1
    [bp_notification] => 1
    [bp_forum_tab] => 1
    [um_profile] => 0
    [um_forum_tab] => 1
    [um_notification] => 1
    [user-synch] => 0
    [role-synch] => 1
    [output-buffer] => 1
    [wp-date-format] => 0
    [subscribe_conf] => 1
    [subscribe_checkbox_on_post_editor] => 1
    [subscribe_checkbox_default_status] => 0
    [attach-media-lib] => 1
    [debug-mode] => 1
    [copyright] => 1
    [profile] => 1
    [notifications] => 1
    [notifications-live] => 0
    [notifications-bar] => 1
    [goto-unread] => 1
    [goto-unread-button] => 0
    [disable_new_user_admin_notification] => 1
    [option_cache] => 1
    [admin-cp] => 1
)
Array
(
    [font_size_forum] => 17
    [font_size_topic] => 16
    [font_size_post_content] => 14
    [custom_css] => #wpforo #wpforo-wrap {
   font-size: 13px; width: 100%; padding:10px 20px; margin:0px;
}

)
Array
(
    [id] => classic
    [name] => Classic
    [version] => 1.0.0
    [description] => Main wpForo Stylesheet
    [author] => gVectors Team
    [url] => http://wpforo.com
    [file] => classic/style.css
    [folder] => classic
    [layouts] => Array
        (
            [1] => Array
                (
                    [id] => 1
                    [name] => Extended
                    [version] => 1.0.0
                    [description] => Extended layout displays one level deeper information in advance.
                    [author] => gVectors Team
                    [url] => http://gvectors.com/
                    [file] => classic/layouts/1/forum.php
                )

[2] => Array
                (
                    [id] => 2
                    [name] => Simplified
                    [version] => 1.0.0
                    [description] => Simplified layout looks simple and clean.
                    [author] => gVectors Team
                    [url] => http://gvectors.com/
                    [file] => classic/layouts/2/forum.php
                )

[3] => Array
                (
                    [id] => 3
                    [name] => QA
                    [version] => 1.0.0
                    [description] => Q&A Layout turns your forum to a powerful question and answer discussion board.
                    [author] => gVectors Team
                    [url] => http://gvectors.com/
                    [file] => classic/layouts/3/forum.php
                )

[4] => Array
                (
                    [id] => 4
                    [name] => Threaded
                    [version] => 1.0.0
                    [description] => Threaded layout turns your forum to a threads list accented on discussion tree view.
                    [author] => gVectors Team
                    [url] => http://gvectors.com/
                    [file] => classic/layouts/4/forum-sub.php
                )

)

[style] => default
    [styles] => Array
        (
            [default] => Array
                (
                    [0] => #000000
                    [1] => #ffffff
                    [2] => #333333
                    [3] => #555555
                    [4] => #666666
                    [5] => #777777
                    [6] => #999999
                    [7] => #cccccc
                    [8] => #e6e6e6
                    [9] => #f5f5f5
                    [10] => #dadada
                    [11] => #659fbe
                    [12] => #43a6df
                    [13] => #72ccfc
                    [14] => #0099cc
                    [15] => #3f7796
                    [16] => #4a8eb3
                    [17] => #dff4ff
                    [20] => #ff812d
                    [30] => #4dca5c
                    [31] => #00a636
                    [32] => #86ba4c
                    [33] => #6fa634
                    [40] => #ff9595
                    [41] => #ff7575
                    [42] => #f46464
                )

[red] => Array
                (
                    [0] => #000000
                    [1] => #ffffff
                    [2] => #333333
                    [3] => #555555
                    [4] => #666666
                    [5] => #777777
                    [6] => #999999
                    [7] => #cccccc
                    [8] => #e6e6e6
                    [9] => #f5f5f5
                    [10] => #dadada
                    [11] => #E0141E
                    [12] => #EE1A26
                    [13] => #FC979C
                    [14] => #E0141E
                    [15] => #99262B
                    [16] => #D61319
                    [17] => #FFF7F7
                    [20] => #30B2A7
                    [30] => #4dca5c
                    [31] => #00a636
                    [32] => #86ba4c
                    [33] => #6fa634
                    [40] => #ff9595
                    [41] => #ff7575
                    [42] => #f46464
                )

[green] => Array
                (
                    [0] => #000000
                    [1] => #ffffff
                    [2] => #333333
                    [3] => #555555
                    [4] => #666666
                    [5] => #777777
                    [6] => #999999
                    [7] => #cccccc
                    [8] => #e6e6e6
                    [9] => #f5f5f5
                    [10] => #dadada
                    [11] => #6EA500
                    [12] => #649E2D
                    [13] => #8DCE0C
                    [14] => #447714
                    [15] => #5A7F10
                    [16] => #6EA500
                    [17] => #F8FCEF
                    [20] => #ff812d
                    [30] => #4dca5c
                    [31] => #00a636
                    [32] => #FF812D
                    [33] => #F47222
                    [40] => #ff9595
                    [41] => #ff7575
                    [42] => #f46464
                )

[orange] => Array
                (
                    [0] => #000000
                    [1] => #ffffff
                    [2] => #333333
                    [3] => #555555
                    [4] => #666666
                    [5] => #777777
                    [6] => #999999
                    [7] => #cccccc
                    [8] => #e6e6e6
                    [9] => #f5f5f5
                    [10] => #dadada
                    [11] => #E0762F
                    [12] => #FF6600
                    [13] => #FC9958
                    [14] => #F26000
                    [15] => #AA4F12
                    [16] => #F26000
                    [17] => #FFF4ED
                    [20] => #ff812d
                    [30] => #4dca5c
                    [31] => #00a636
                    [32] => #86ba4c
                    [33] => #6fa634
                    [40] => #ff9595
                    [41] => #ff7575
                    [42] => #f46464
                )

[grey] => Array
                (
                    [0] => #000000
                    [1] => #ffffff
                    [2] => #333333
                    [3] => #343434
                    [4] => #666666
                    [5] => #777777
                    [6] => #999999
                    [7] => #cccccc
                    [8] => #e6e6e6
                    [9] => #f5f5f5
                    [10] => #dadada
                    [11] => #888888
                    [12] => #666666
                    [13] => #7EEA8D
                    [14] => #777777
                    [15] => #333333
                    [16] => #555555
                    [17] => #DFF4FF
                    [20] => #FF812D
                    [30] => #4dca5c
                    [31] => #00a636
                    [32] => #86ba4c
                    [33] => #6fa634
                    [40] => #ff9595
                    [41] => #ff7575
                    [42] => #f46464
                )

[dark] => Array
                (
                    [0] => #000000
                    [1] => #141414
                    [2] => #bbbbbb
                    [3] => #000000
                    [4] => #666666
                    [5] => #bcbcbc
                    [6] => #999999
                    [7] => #585858
                    [8] => #727272
                    [9] => #323232
                    [10] => #dadada
                    [11] => #888888
                    [12] => #33779b
                    [13] => #7EEA8D
                    [14] => #777777
                    [15] => #E0E0E0
                    [16] => #CECECE
                    [17] => #33779b
                    [20] => #FF812D
                    [30] => #4dca5c
                    [31] => #00a636
                    [32] => #86ba4c
                    [33] => #6fa634
                    [40] => #ff9595
                    [41] => #ff7575
                    [42] => #f46464
                )

)

)
classic