It is currently Tue, 14 Jul 2020 00:38:17 GMT



 
Author Message
 kernel meltdown
Hi there,
        I am hoping someone can help me.  This morning one of our web servers crapped itself and I don't know why.  It's running RedHat 7.2 kernel version 2.4.9-31smp.  I couldn't login from the console so I had to reset it.  When it came back up it was fine.  This is what I am seeing in the messages log:

Jun  7 02:57:43 web02 kernel: kernel BUG at slab.c:1767!
Jun  7 02:57:43 web02 kernel: invalid operand: 0000
Jun  7 02:57:43 web02 kernel: Kernel 2.4.9-31smp
Jun  7 02:57:43 web02 kernel: CPU:    0
Jun  7 02:57:43 web02 kernel: EIP:    0010:[kmem_cache_reap+504/912]    Not tainted
Jun  7 02:57:43 web02 kernel: EIP:    0010:[<c0133d08>]    Not tainted
Jun  7 02:57:43 web02 kernel: EFLAGS: 00010092
Jun  7 02:57:43 web02 kernel: EIP is at kmem_cache_reap [kernel] 0x1f8
Jun  7 02:57:43 web02 kernel: eax: 0000001b   ebx: e2c34000   ecx: c02db9e4   edx: 00003906
Jun  7 02:57:43 web02 kernel: esi: c1b8f9e8   edi: c1b8f9f8   ebp: 00000000   esp: e3fedf8c
Jun  7 02:57:43 web02 kernel: ds: 0018   es: 0018   ss: 0018
Jun  7 02:57:43 web02 kernel: Process kswapd (pid: 5, stackpage=e3fed000)
Jun  7 02:57:43 web02 kernel: Stack: c024f253 000006e7 00000d80 c1b8f9f8 c1b8f9f0 00000183 e3f9d000 0000000a
Jun  7 02:57:43 web02 kernel:        00000000 00000000 00000000 00000183 000000c0 000000c0 0008e000 c0136076
Jun  7 02:57:43 web02 kernel:        000000c0 e3fec000 00000006 c01360d5 000000c0 00000000 00010f00 c1d9ffb8
Jun  7 02:57:43 web02 kernel: Call Trace: [call_spurious_interrupt+130654/156203] .rodata.str1.1 [kernel] 0x2a8e
Jun  7 02:57:43 web02 kernel: Call Trace: [<c024f253>] .rodata.str1.1 [kernel] 0x2a8e
Jun  7 02:57:43 web02 kernel: [do_try_to_free_pages+70/80] do_try_to_free_pages [kernel] 0x46
Jun  7 02:57:43 web02 kernel: [<c0136076>] do_try_to_free_pages [kernel] 0x46
Jun  7 02:57:43 web02 kernel: [kswapd+85/240] kswapd [kernel] 0x55
Jun  7 02:57:43 web02 kernel: [<c01360d5>] kswapd [kernel] 0x55
Jun  7 02:57:43 web02 kernel: [_stext+0/96] stext [kernel] 0x0
Jun  7 02:57:43 web02 kernel: [<c0105000>] stext [kernel] 0x0
Jun  7 02:57:43 web02 kernel: [kernel_thread+38/48] kernel_thread [kernel] 0x26
Jun  7 02:57:43 web02 kernel: [<c0105866>] kernel_thread [kernel] 0x26
Jun  7 02:57:43 web02 kernel: [kswapd+0/240] kswapd [kernel] 0x0
Jun  7 02:57:43 web02 kernel: [<c0136080>] kswapd [kernel] 0x0
Jun  7 02:57:43 web02 kernel:
Jun  7 02:57:43 web02 kernel:
Jun  7 02:57:43 web02 kernel: Code: 0f 0b 58 5a 8b 03 45 39 f8 75 dd 8b 4e 2c 89 ea 8b 7e 4c d3

I have searched on google for some of these messages but I couldn't find anything helpful.  Any help or direction would be apprecaited.

I am not subscribed to this list so email to me directly would be great.  If I am mailing to the wrong list my apologies.

Thanks so much all!

-anna
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at   http://www.**-**.com/
Please read the FAQ at   http://www.**-**.com/



 Wed, 24 Nov 2004 01:50:07 GMT   
 kernel meltdown

Here is the output and thanks Thunder:

ksymoops 2.4.1 on i686 2.4.9-31smp.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.9-31smp/ (default)
     -m /boot/System.map-2.4.9-31smp (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

Error (expand_objects): cannot stat(/lib/ext3.o) for ext3
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/jbd.o) for jbd
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/sym53c8xx.o) for sym53c8xx
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
ksymoops: No such file or directory
/usr/bin/find: /lib/modules/2.4.9-31smp/build: No such file or directory
Error (pclose_local): find_objects pclose failed 0x100

Warning (compare_maps): ksyms_base symbol
GPLONLY_IO_APIC_get_PCI_irq_vector not found in System.map.  Ignoring
ksyms_base entry
Warning (compare_maps): ksyms_base symbol
GPLONLY_pci_hp_change_slot_info not found in System.map.  Ignoring
ksyms_base entry
Warning (compare_maps): ksyms_base symbol GPLONLY_pci_hp_deregister not
found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol GPLONLY_pci_hp_register not
found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol partition_name  , ksyms_base
says c01c2f40, System.map says c0161e20.  Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol nlmsvc_grace_period  , lockd
says e4924e54, /lib/modules/2.4.9-31smp/kernel/fs/lockd/lockd.o says
e49242b4.  Ignoring /lib/modules/2.4.9-31smp/kernel/fs/lockd/lockd.o
entry
Warning (compare_maps): mismatch on symbol nlmsvc_ops  , lockd says
e4924e50, /lib/modules/2.4.9-31smp/kernel/fs/lockd/lockd.o says
e49242b0.  Ignoring /lib/modules/2.4.9-31smp/kernel/fs/lockd/lockd.o
entry
Warning (compare_maps): mismatch on symbol nlmsvc_timeout  , lockd says
e4924e58, /lib/modules/2.4.9-31smp/kernel/fs/lockd/lockd.o says
e49242b8.  Ignoring /lib/modules/2.4.9-31smp/kernel/fs/lockd/lockd.o
entry
Warning (compare_maps): mismatch on symbol nfs_debug  , sunrpc says
e4916180, /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o says
e4915e60.  Ignoring /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o
entry
Warning (compare_maps): mismatch on symbol nfsd_debug  , sunrpc says
e4916184, /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o says
e4915e64.  Ignoring /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o
entry
Warning (compare_maps): mismatch on symbol nlm_debug  , sunrpc says
e4916188, /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o says
e4915e68.  Ignoring /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o
entry
Warning (compare_maps): mismatch on symbol rpc_debug  , sunrpc says
e491617c, /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o says
e4915e5c.  Ignoring /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o
entry
Warning (compare_maps): mismatch on symbol rpc_garbage_args  , sunrpc
says e491615c, /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o says
e4915e3c.  Ignoring /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o
entry
Warning (compare_maps): mismatch on symbol rpc_success  , sunrpc says
e491614c, /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o says
e4915e2c.  Ignoring /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o
entry
Warning (compare_maps): mismatch on symbol rpc_system_err  , sunrpc says
e4916160, /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o says
e4915e40.  Ignoring /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o
entry
Warning (compare_maps): mismatch on symbol xdr_one  , sunrpc says
e4916144, /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o says
e4915e24.  Ignoring /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o
entry
Warning (compare_maps): mismatch on symbol xdr_two  , sunrpc says
e4916148, /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o says
e4915e28.  Ignoring /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o
entry
Warning (compare_maps): mismatch on symbol xdr_zero  , sunrpc says
e4916140, /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o says
e4915e20.  Ignoring /lib/modules/2.4.9-31smp/kernel/net/sunrpc/sunrpc.o
entry
Warning (compare_maps): mismatch on symbol usb_devfs_handle  , usbcore
says e48db520, /lib/modules/2.4.9-31smp/kernel/drivers/usb/usbcore.o
says e48db040.  Ignoring
/lib/modules/2.4.9-31smp/kernel/drivers/usb/usbcore.o entry
Warning (map_ksym_to_module): cannot match loaded module ext3 to a
unique module object.  Trace may not be reliable.
Warning (compare_maps): mismatch on symbol sd  , sd_mod says e481ce60,
/lib/modules/2.4.9-31smp/kernel/drivers/scsi/sd_mod.o says e481cdc0.
Ignoring /lib/modules/2.4.9-31smp/kernel/drivers/scsi/sd_mod.o entry
Warning (compare_maps): mismatch on symbol proc_scsi  , scsi_mod says
e4818c9c, /lib/modules/2.4.9-31smp/kernel/drivers/scsi/scsi_mod.o says
e48174d4.  Ignoring
/lib/modules/2.4.9-31smp/kernel/drivers/scsi/scsi_mod.o entry
Warning (compare_maps): mismatch on symbol scsi_devicelist  , scsi_mod
says e4818cc8, /lib/modules/2.4.9-31smp/kernel/drivers/scsi/scsi_mod.o
says e4817500.  Ignoring
/lib/modules/2.4.9-31smp/kernel/drivers/scsi/scsi_mod.o entry
Warning (compare_maps): mismatch on symbol scsi_hostlist  , scsi_mod
says e4818cc4, /lib/modules/2.4.9-31smp/kernel/drivers/scsi/scsi_mod.o
says e48174fc.  Ignoring
/lib/modules/2.4.9-31smp/kernel/drivers/scsi/scsi_mod.o entry
Warning (compare_maps): mismatch on symbol scsi_hosts  , scsi_mod says
e4818ccc, /lib/modules/2.4.9-31smp/kernel/drivers/scsi/scsi_mod.o says
e4817504.  Ignoring
/lib/modules/2.4.9-31smp/kernel/drivers/scsi/scsi_mod.o entry
Warning (compare_maps): mismatch on symbol scsi_logging_level  ,
scsi_mod says e4818c98,
/lib/modules/2.4.9-31smp/kernel/drivers/scsi/scsi_mod.o says e48174d0.
Ignoring /lib/modules/2.4.9-31smp/kernel/drivers/scsi/scsi_mod.o entry
 kernel BUG at slab.c:1767!
 invalid operand: 0000
 CPU:    0
 EIP:    0010:[kmem_cache_reap+504/912]    Not tainted
 EIP:    0010:[<c0133d08>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
 EFLAGS: 00010092
 eax: 0000001b   ebx: e2c34000   ecx: c02db9e4   edx: 00003906
 esi: c1b8f9e8   edi: c1b8f9f8   ebp: 00000000   esp: e3fedf8c
 ds: 0018   es: 0018   ss: 0018
 Process kswapd (pid: 5, stackpage=e3fed000)
 Stack: c024f253 000006e7 00000d80 c1b8f9f8 c1b8f9f0 00000183 e3f9d000
0000000a
        00000000 00000000 00000000 00000183 000000c0 000000c0 0008e000
c0136076
        000000c0 e3fec000 00000006 c01360d5 000000c0 00000000 00010f00
c1d9ffb8
 Call Trace: [call_spurious_interrupt+130654/156203] .rodata.str1.1
[kernel] 0x2a8e
 Call Trace: [<c024f253>] .rodata.str1.1 [kernel] 0x2a8e
 [<c0136076>] do_try_to_free_pages [kernel] 0x46
 [<c01360d5>] kswapd [kernel] 0x55
 [<c0105000>] stext [kernel] 0x0
 [<c0105866>] kernel_thread [kernel] 0x26
 [<c0136080>] kswapd [kernel] 0x0
 Code: 0f 0b 58 5a 8b 03 45 39 f8 75 dd 8b 4e 2c 89 ea 8b 7e 4c d3

Trace; c024f253 <call_spurious_interrupt+1fe5e/2622b>
Trace; c0136076 <do_try_to_free_pages+46/50>
Trace; c01360d5 <kswapd+55/f0>
Trace; c0105000 <_stext+0/0>
Trace; c0105866 <kernel_thread+26/30>
Trace; c0136080 <kswapd+0/f0>
Code;  c0133d08 <kmem_cache_reap+1f8/390>
00000000 <_EIP>:
Code;  c0133d08 <kmem_cache_reap+1f8/390>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c0133d0a <kmem_cache_reap+1fa/390>
   2:   58                        pop    %eax
Code;  c0133d0b <kmem_cache_reap+1fb/390>
   3:   5a                        pop    %edx
Code;  c0133d0c <kmem_cache_reap+1fc/390>
   4:   8b 03                     mov    (%ebx),%eax
Code;  c0133d0e <kmem_cache_reap+1fe/390>
   6:   45                        inc    %ebp
Code;  c0133d0f <kmem_cache_reap+1ff/390>
   7:   39 f8                     cmp    %edi,%eax
Code;  c0133d11 <kmem_cache_reap+201/390>
   9:   75 dd                     jne    ffffffe8 <_EIP+0xffffffe8>
c0133cf0 <kmem_cache_reap+1e0/390>
Code;  c0133d13 <kmem_cache_reap+203/390>
   b:   8b 4e 2c                  mov    0x2c(%esi),%ecx
Code;  c0133d16 <kmem_cache_reap+206/390>
   e:   89 ea                     mov    %ebp,%edx
Code;  c0133d18 <kmem_cache_reap+208/390>
  10:   8b 7e 4c                  mov    0x4c(%esi),%edi
Code;  c0133d1b <kmem_cache_reap+20b/390>
  13:   d3 00                     roll   %cl,(%eax)

27 warnings and 6 errors issued.  Results may not be reliable.

-anna
...

read more »



 Wed, 24 Nov 2004 02:40:07 GMT   
 kernel meltdown

I looked up that line in the source and found this piece of code:

                full_free = 0;
                p = searchp->slabs_free.next;
                while (p != &searchp->slabs_free) {
                        slabp = list_entry(p, slab_t, list);
                        if (slabp->inuse)
                                BUG();
                        full_free++;
                        p = p->next;
                }

Could it be a race with this particular slabp being taken in use
by another CPU at this very moment?

--
Kasper Dupont -- der bruger for meget tid p? usenet.
For sending spam use mailto:razor-rep...@daimi.au.dk
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



 Wed, 24 Nov 2004 07:03:27 GMT   
 kernel meltdown
RedHat have released an updated kernel (2.4.9-34).

Does this help?

---------------------------------------------
Phil Randal
Network Engineer
Herefordshire Council
Hereford, UK

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



 Fri, 26 Nov 2004 17:20:10 GMT   
 
   [ 4 post ] 

Similar Threads

1. e-mail server problems after kernel meltdown

2. Kernel 2.4.7 - Boot Meltdown

3. need help with baffling eth0 meltdown

4. Meltdown with Allied Telesyn AT-2450

5. meltdown source code

6. meltdown bin?

7. linux boot meltdown!!

8. Hard drive meltdown

9. Major Meltdown with Linux SCSI!


 
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by ST Software