Администрирую несколько веб-серверов на Gentoo. На днях ни с того ни с сего приключился kernel panic.
Вот кусок dmesg по этому поводу:
Aug 3 03:10:01 fastvps0 kernel: [4215962.078076] general protection fault: 0000 [#1] SMP Aug 3 03:10:01 fastvps0 kernel: [4215962.078248] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.3/i2c-0/name Aug 3 03:10:01 fastvps0 kernel: [4215962.078567] CPU 3 Aug 3 03:10:01 fastvps0 kernel: [4215962.078574] Modules linked in: tun dm_snapshot scsi_wait_scan Aug 3 03:10:01 fastvps0 kernel: [4215962.078903] Aug 3 03:10:01 fastvps0 kernel: [4215962.079063] Pid: 30230, comm: apache2 Not tainted 2.6.36-gentoo-r5 #1 MSI X58 Pro-E (MS-7522)/MS-7522 Aug 3 03:10:01 fastvps0 kernel: [4215962.079388] RIP: 0010:[<ffffffff810310a7>] [<ffffffff810310a7>] task_rq_lock+0x22/0x6b Aug 3 03:10:01 fastvps0 kernel: [4215962.079718] RSP: 0018:ffff8802bf53fdd8 EFLAGS: 00010046 Aug 3 03:10:01 fastvps0 kernel: [4215962.079883] RAX: 0000000000000246 RBX: 0000000000012d80 RCX: 0000000000000057 Aug 3 03:10:01 fastvps0 kernel: [4215962.080204] RDX: 0000000000000000 RSI: ffff8802bf53fe20 RDI: 2e66c3f300459a40 Aug 3 03:10:01 fastvps0 kernel: [4215962.080525] RBP: ffff8802bf53fdf8 R08: ffff8802bf53fd88 R09: 00000002bf53fde8 Aug 3 03:10:01 fastvps0 kernel: [4215962.080846] R10: 0000000000000400 R11: ffffffff81b802b0 R12: 0000000000012d80 Aug 3 03:10:01 fastvps0 kernel: [4215962.081166] R13: ffff8802bf53fe20 R14: 2e66c3f300459a40 R15: 0000000000000003 Aug 3 03:10:01 fastvps0 kernel: [4215962.081486] FS: 00007f3ef36c5740(0000) GS:ffff880001ec0000(0000) knlGS:0000000000000000 Aug 3 03:10:01 fastvps0 kernel: [4215962.081808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 3 03:10:01 fastvps0 kernel: [4215962.081975] CR2: 0000000001040310 CR3: 0000000261f49000 CR4: 00000000000006e0 Aug 3 03:10:01 fastvps0 kernel: [4215962.082296] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 3 03:10:01 fastvps0 kernel: [4215962.082617] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 3 03:10:01 fastvps0 kernel: [4215962.082938] Process apache2 (pid: 30230, threadinfo ffff8802bf53e000, task ffff88027a4e6740) Aug 3 03:10:01 fastvps0 kernel: [4215962.083261] Stack: Aug 3 03:10:01 fastvps0 kernel: [4215962.083421] 2e66c3f300459a40 ffffffff81a25720 0000000000000000 000000000000000f Aug 3 03:10:01 fastvps0 kernel: [4215962.083596] <0> ffff8802bf53fe58 ffffffff810379bf 0000000000000005 0000000081a25750 Aug 3 03:10:01 fastvps0 kernel: [4215962.083926] <0> ffff8802bf53fe48 0000000000000246 ffff88000730ccd0 ffff88000730ccd0 Aug 3 03:10:01 fastvps0 kernel: [4215962.084409] Call Trace: Aug 3 03:10:01 fastvps0 kernel: [4215962.084574] [<ffffffff810379bf>] try_to_wake_up+0x2f/0x220 Aug 3 03:10:01 fastvps0 kernel: [4215962.084740] [<ffffffff81037bdc>] wake_up_process+0x10/0x12 Aug 3 03:10:01 fastvps0 kernel: [4215962.084907] [<ffffffff81197204>] freeary+0x1a9/0x1e6 Aug 3 03:10:01 fastvps0 kernel: [4215962.085071] [<ffffffff8119755f>] T.503+0x8e/0xc4 Aug 3 03:10:01 fastvps0 kernel: [4215962.085238] [<ffffffff8150aeff>] ? do_page_fault+0x25d/0x28a Aug 3 03:10:01 fastvps0 kernel: [4215962.085405] [<ffffffff81197caa>] sys_semctl+0x74/0x81 Aug 3 03:10:01 fastvps0 kernel: [4215962.085574] [<ffffffff810029eb>] system_call_fastpath+0x16/0x1b Aug 3 03:10:01 fastvps0 kernel: [4215962.085740] Code: 26 6c 4d 00 5b 41 5c c9 c3 55 48 89 e5 41 56 49 89 fe 41 55 49 89 f5 41 54 49 c7 c4 80 2d 01 00 53 9c 58 fa 49 89 45 00 4c 89 e3 <49> 8b 46 08 8b 40 18 48 03 1c c5 10 fe ac 81 48 89 df e8 e8 6b Aug 3 03:10:01 fastvps0 kernel: [4215962.086390] RIP [<ffffffff810310a7>] task_rq_lock+0x22/0x6b Aug 3 03:10:01 fastvps0 kernel: [4215962.086563] RSP <ffff8802bf53fdd8> Aug 3 03:10:01 fastvps0 kernel: [4215962.086981] ---[ end trace 5c52f3a7773aa044 ]--- Aug 3 03:10:01 fastvps0 kernel: [4215962.086985] BUG: scheduling while atomic: apache2/1643/0x10000001 Aug 3 03:10:01 fastvps0 kernel: [4215962.086988] Modules linked in: tun dm_snapshot scsi_wait_scan Aug 3 03:10:01 fastvps0 kernel: [4215962.086992] Pid: 1643, comm: apache2 Tainted: G D 2.6.36-gentoo-r5 #1 Aug 3 03:10:01 fastvps0 kernel: [4215962.086993] Call Trace: Aug 3 03:10:01 fastvps0 kernel: [4215962.086994] <#SS> [<ffffffff81035289>] __schedule_bug+0x57/0x5b Aug 3 03:10:01 fastvps0 kernel: [4215962.087004] [<ffffffff81506424>] schedule+0xe6/0x64a Aug 3 03:10:01 fastvps0 kernel: [4215962.087008] [<ffffffff8150aef6>] ? do_page_fault+0x254/0x28a Aug 3 03:10:01 fastvps0 kernel: [4215962.087011] [<ffffffff81038aa8>] __cond_resched+0x25/0x30 Aug 3 03:10:01 fastvps0 kernel: [4215962.087012] [<ffffffff8150aef6>] ? do_page_fault+0x254/0x28a Aug 3 03:10:01 fastvps0 kernel: [4215962.087015] [<ffffffff81506aa9>] _cond_resched+0x27/0x32 Aug 3 03:10:01 fastvps0 kernel: [4215962.087018] [<ffffffff81004e45>] is_valid_bugaddr+0x11/0x2a Aug 3 03:10:01 fastvps0 kernel: [4215962.087021] [<ffffffff811daf35>] report_bug+0x1c/0xe0 Aug 3 03:10:01 fastvps0 kernel: [4215962.087023] [<ffffffff81005dd8>] die+0x34/0x5e Aug 3 03:10:01 fastvps0 kernel: [4215962.087025] [<ffffffff81508885>] do_trap+0x11c/0x12b Aug 3 03:10:01 fastvps0 kernel: [4215962.087027] [<ffffffff81003c75>] do_stack_segment+0x6f/0x8d Aug 3 03:10:01 fastvps0 kernel: [4215962.087029] [<ffffffff815082ff>] stack_segment+0x1f/0x30 Aug 3 03:10:01 fastvps0 kernel: [4215962.087031] [<ffffffff8150aef6>] ? do_page_fault+0x254/0x28a Aug 3 03:10:01 fastvps0 kernel: [4215962.087032] <<EOE>> [<ffffffff8150835f>] ? page_fault+0x1f/0x30 Aug 3 03:10:01 fastvps0 kernel: [4215962.087044] stack segment: 0000 [#2] SMP Aug 3 03:10:01 fastvps0 kernel: [4215962.087046] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.3/i2c-0/name Aug 3 03:10:01 fastvps0 kernel: [4215962.087048] CPU 1 Aug 3 03:10:01 fastvps0 kernel: [4215962.087048] Modules linked in: tun dm_snapshot scsi_wait_scan Aug 3 03:10:01 fastvps0 kernel: [4215962.087050] Aug 3 03:10:01 fastvps0 kernel: [4215962.087052] Pid: 1643, comm: apache2 Tainted: G D 2.6.36-gentoo-r5 #1 MSI X58 Pro-E (MS-7522)/MS-7522 Aug 3 03:10:01 fastvps0 kernel: [4215962.087054] RIP: 0010:[<ffffffff8150aef6>] [<ffffffff8150aef6>] do_page_fault+0x254/0x28a Aug 3 03:10:01 fastvps0 kernel: [4215962.087057] RSP: 0000:ffff88032a933f08 EFLAGS: 00010246 Aug 3 03:10:01 fastvps0 kernel: [4215962.087058] RAX: 0000000000000000 RBX: 0000000001006788 RCX: 0000000000000001 Aug 3 03:10:01 fastvps0 kernel: [4215962.087059] RDX: 0000000001006788 RSI: ffff88032a933f58 RDI: 0000000000000005 Aug 3 03:10:01 fastvps0 kernel: [4215962.087061] RBP: 2a933f482a933f48 R08: 0000000001006788 R09: 0000000000000005 Aug 3 03:10:01 fastvps0 kernel: [4215962.087062] R10: ffffea0009b2bcf8 R11: ffff88033d306180 R12: ffff88032a933f58 Aug 3 03:10:01 fastvps0 kernel: [4215962.087063] R13: 0000000000000007 R14: ffff8800069ea170 R15: ffff8802e9f7e740 Aug 3 03:10:01 fastvps0 kernel: [4215962.087065] FS: 00007f3e9c5fe710(0000) GS:ffff880001e40000(0000) knlGS:0000000000000000 Aug 3 03:10:01 fastvps0 kernel: [4215962.087067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 3 03:10:01 fastvps0 kernel: [4215962.087068] CR2: 0000000001006788 CR3: 00000002bd032000 CR4: 00000000000006e0 Aug 3 03:10:01 fastvps0 kernel: [4215962.087070] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 3 03:10:01 fastvps0 kernel: [4215962.087071] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 3 03:10:01 fastvps0 kernel: [4215962.087073] Process apache2 (pid: 1643, threadinfo ffff88032a932000, task ffff8802e9f7e740) Aug 3 03:10:01 fastvps0 kernel: [4215962.087074] Stack: Aug 3 03:10:01 fastvps0 kernel: [4215962.087075] ffff88033d3061e0 ffff88033d306180 ffff88032a933f48 0000000000000000 Aug 3 03:10:01 fastvps0 kernel: [4215962.087076] <0> 00007f3e9c5fde6c 0000000000000001 00007f3e9c5fde60 0000000000000080 Aug 3 03:10:01 fastvps0 kernel: [4215962.087078] <0> 0000000001006490 ffffffff8150835f 0000000000000080 00007f3e9c5fde60 Aug 3 03:10:01 fastvps0 kernel: [4215962.087080] Call Trace: Aug 3 03:10:01 fastvps0 kernel: [4215962.087082] [<ffffffff8150835f>] ? page_fault+0x1f/0x30 Aug 3 03:10:01 fastvps0 kernel: [4215962.087083] Code: 49 ff 87 a8 03 00 00 48 89 da 4c 89 e6 bf 06 00 00 00 eb 12 49 ff 87 a0 03 00 00 48 89 da 4c 89 e6 bf 05 00 00 00 e8 26 8b b1 ff <48> 8b 7d c0 e8 e1 da b4 ff eb 1c 31 c9 4c 89 f6 85 d2 48 8b 7d Aug 3 03:10:01 fastvps0 kernel: [4215962.087094] RIP [<ffffffff8150aef6>] do_page_fault+0x254/0x28a Aug 3 03:10:01 fastvps0 kernel: [4215962.087096] RSP <ffff88032a933f08>
В чем может быть проблема? Был ли апач причиной падения или жертвой? Что предпринять, чтобы собрать больше информации?
З.Ы. По поводу железа хостер уверяет, что все исправно.
Апач не может быть проблемой
Апач не может быть причиной по-определению. Проблема в ядре.
(*)
https://lkml.org/lkml/2011/7/19/245
https://bugzilla.kernel.org/show_bug.cgi?id=27142
Искал так - взял три строки после "Call Trace" и очистил от "шелухи".
Получил try_to_wake_up wake_up_process freeary
Это и гуглил.
lspci | grep 00:1fпокажи
lspci | grep 00:1f
покажи плиз ...
gentoo на kvm guest?
Вот вывод 00:1f.0 ISA
Вот вывод
Система стоит на реальном железе.
Сохранить мир подчас труднее, чем выстоять в самой жестокой схватке...
Не верь хостеру - проблема
Не верь хостеру - проблема скорее всего (90%) аппаратная.
Это так называемый CPU exception, почитать можно тут: http://wiki.osdev.org/Exceptions#General_Protection_Fault
Вы назвали последствие, но не
Вы назвали последствие, но не назвали причину. 90% случаев у хостеров - это тупые проблемы юзверей и 10% это то что они сами СЗБ (пытаются по макимому использовать ресурсы железа накидав туда побольше пользователей).
Хардварный exception из-за
Хардварный exception из-за юзера? оО