You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-02 - 05:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Jul 02, 2026 00:45:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
355878899119115,2cyclictest0-21swapper/019:15:020
3558793998623,19cyclictest275-21dbus-daemon00:25:011
3558793998623,19cyclictest275-21dbus-daemon00:25:001
3558788998270,3cyclictest0-21swapper/020:00:010
3558802998171,7cyclictest37-21ksoftirqd/321:20:023
3558797997742,6cyclictest3549852-21kworker/2:122:22:512
3558793997020,19cyclictest3646220-40mandb00:00:031
3558793997011,18cyclictest3644306-21ssh23:55:311
355878899704,31cyclictest131rcu_preempt00:15:260
355878899704,31cyclictest131rcu_preempt00:15:250
3558793996916,20cyclictest3640851-21ssh23:48:351
355880299671,49cyclictest3467602-21systemd-journal21:50:023
355879799665,55cyclictest3563214-21wc20:35:022
3558797996625,37cyclictest3562807-21sed20:25:022
3558793996610,20cyclictest3655321-21ssh00:18:551
3558793996610,20cyclictest3655321-21ssh00:18:551
355878899667,32cyclictest364-21snmpd00:05:020
355878899663,9cyclictest121ksoftirqd/022:19:570
355880299659,13cyclictest131rcu_preempt23:13:473
355880299659,13cyclictest131rcu_preempt23:13:473
3558797996520,11cyclictest30-21ksoftirqd/221:25:002
355878899658,29cyclictest131rcu_preempt23:53:370
355878899656,6cyclictest131rcu_preempt23:20:000
3558802996455,6cyclictest37-21ksoftirqd/323:52:473
3558797996426,33cyclictest3654049-21sh00:16:082
3558797996426,33cyclictest3654049-21sh00:16:072
3558797996423,13cyclictest30-21ksoftirqd/222:16:452
3558797996420,5cyclictest3627455-21sh23:20:042
3558797996420,5cyclictest3627455-21sh23:20:032
355879399647,21cyclictest3615868-21rm22:55:541
3558793996410,17cyclictest3622853-21ssh23:10:221
3558793996410,17cyclictest3622853-21ssh23:10:221
3558793996317,9cyclictest23-21ksoftirqd/123:33:201
3558793996310,13cyclictest3577503-21ssh21:35:541
3558788996331,9cyclictest12-21ksoftirqd/023:03:540
3558788996330,30cyclictest12-21ksoftirqd/020:40:010
3558797996227,31cyclictest30-21ksoftirqd/220:59:032
3558797996226,31cyclictest3632507-21sh23:30:412
3558797996225,33cyclictest3596362-21latency_hist22:15:022
3558797996222,11cyclictest30-21ksoftirqd/222:34:512
3558797996219,7cyclictest3632834-1kworker/u9:123:36:312
3558797996219,11cyclictest30-21ksoftirqd/221:25:532
355879399626,17cyclictest1-21systemd22:20:011
355878899623,54cyclictest3609932-21ssh22:43:450
3558797996152,6cyclictest3633360-21kworker/2:223:45:022
3558797996125,12cyclictest275-21dbus-daemon20:15:012
3558797996120,11cyclictest30-21ksoftirqd/221:50:302
355879399619,16cyclictest3631126-21ssh23:28:051
3558793996116,7cyclictest23-21ksoftirqd/121:41:311
3558793996114,15cyclictest3565406-21ssh21:10:351
3558793996111,15cyclictest3600390-21ssh22:23:511
3558788996131,15cyclictest3589294-21ssh22:00:260
3558797996030,26cyclictest3618944-21sh23:02:222
3558797996024,32cyclictest3608131-21rm22:40:022
3558797996021,12cyclictest30-21ksoftirqd/221:55:162
3558797996021,12cyclictest30-21ksoftirqd/221:55:162
3558797996018,10cyclictest3564213-21sshd20:52:032
355879399609,15cyclictest3559341-21cat19:20:021
355879399607,15cyclictest3626530-21ssh23:18:281
3558793996018,8cyclictest23-21ksoftirqd/122:35:001
3558793996015,9cyclictest23-21ksoftirqd/123:24:161
3558793996015,9cyclictest23-21ksoftirqd/123:24:161
3558793996012,14cyclictest3570212-21ssh21:20:371
355879799593,21cyclictest3610454-21munin-run22:45:002
3558797995924,11cyclictest30-21ksoftirqd/200:38:482
3558797995923,11cyclictest30-21ksoftirqd/223:15:182
3558797995920,5cyclictest3563608-21latency_hist20:40:022
3558797995919,7cyclictest30-21ksoftirqd/223:53:132
3558797995917,5cyclictest3622645-21latency_hist23:10:022
3558797995917,10cyclictest1-21systemd00:25:012
3558797995917,10cyclictest1-21systemd00:25:002
355879399597,15cyclictest3595333-21ssh22:13:111
355879399596,15cyclictest3572685-21ssh21:25:571
355879399594,16cyclictest3574915-21ssh21:30:361
355879399594,16cyclictest3574915-21ssh21:30:361
3558793995917,11cyclictest23-21ksoftirqd/100:37:331
3558793995916,15cyclictest3643769-21ssh23:54:531
3558793995915,13cyclictest3608515-21ssh22:40:261
3558788995931,12cyclictest3616811-21ssh22:57:590
3558788995928,15cyclictest3630442-21ssh23:26:310
355879799583,19cyclictest311-21rs:main2
3558797995821,5cyclictest3624381-21sh23:14:002
3558797995821,5cyclictest3624381-21sh23:13:592
3558797995819,9cyclictest30-21ksoftirqd/222:47:512
3558793995815,7cyclictest23-21ksoftirqd/100:07:091
3558788995827,26cyclictest1-21systemd19:55:010
355878899582,39cyclictest3622677-21sed23:10:020
3558797995721,11cyclictest309-21in:imuxsock21:45:002
3558797995717,9cyclictest30-21ksoftirqd/200:33:472
3558797995715,11cyclictest3569682-21cron21:20:002
355879399578,14cyclictest3563698-21apt-config20:40:351
3558788995733,19cyclictest12-21ksoftirqd/022:25:030
3558788995729,13cyclictest3594500-21ssh22:11:200
355878899572,7cyclictest3565036-21modprobe21:10:000
3558788995726,15cyclictest3639573-21ssh23:45:350
3558788995726,15cyclictest3579321-21ssh21:40:010
3558788995713,23cyclictest12-21ksoftirqd/022:10:020
3558797995634,13cyclictest3467602-21systemd-journal20:00:002
3558797995623,6cyclictest30-21ksoftirqd/222:09:592
3558797995622,4cyclictest3559637-21latency_hist19:25:012
3558797995621,31cyclictest0-21swapper/221:15:022
3558797995619,8cyclictest30-21ksoftirqd/221:49:002
355879399568,18cyclictest3561810-21date20:05:031
355879399568,16cyclictest3608106-21munin-run22:40:021
355879399567,16cyclictest3565025-21diskmemload00:31:091
3558788995628,12cyclictest3637404-21ssh23:41:000
3558788995625,15cyclictest3632661-21ssh23:31:020
3558797995538,13cyclictest1-21systemd20:40:332
355879799553,16cyclictest1-21systemd23:00:012
3558797995522,8cyclictest30-21ksoftirqd/223:59:512
355878899559,5cyclictest131rcu_preempt22:39:510
3558788995545,6cyclictest151rcuc/000:25:000
3558788995545,6cyclictest151rcuc/000:25:000
3558788995538,12cyclictest275-21dbus-daemon20:55:010
3558788995527,13cyclictest3628168-21ssh23:21:470
3558788995527,13cyclictest3628168-21ssh23:21:470
3558788995526,14cyclictest3623637-21ssh23:12:170
3558788995526,14cyclictest3623637-21ssh23:12:170
3558788995524,15cyclictest3585243-21ssh21:52:050
3558788995520,17cyclictest3651009-21ssh00:10:010
3558788995518,32cyclictest0-21swapper/000:00:000
3558797995447,3cyclictest30-21ksoftirqd/219:40:022
3558797995445,6cyclictest3588892-21kworker/2:022:04:462
3558797995419,4cyclictest3615137-21sh22:54:412
3558797995411,9cyclictest0-21swapper/200:10:012
355879399549,15cyclictest3562690-21sed20:25:011
3558793995414,10cyclictest23-21ksoftirqd/121:50:311
3558793995413,8cyclictest23-21ksoftirqd/123:09:401
355878899543,12cyclictest0-21swapper/021:20:000
3558788995426,13cyclictest3604852-21ssh22:33:030
355880299535,38cyclictest3569941-21rm21:20:033
3558797995323,5cyclictest30-21ksoftirqd/222:26:262
3558797995322,4cyclictest3648289-21sh00:04:162
355879799532,15cyclictest3559915-21wc19:30:022
3558797995317,7cyclictest30-21ksoftirqd/220:10:022
355879399534,11cyclictest3565025-21diskmemload22:54:291
3558793995335,14cyclictest3467602-21systemd-journal20:40:001
3558793995315,8cyclictest23-21ksoftirqd/121:46:121
3558793995314,9cyclictest23-21ksoftirqd/122:02:031
3558793995313,7cyclictest23-21ksoftirqd/100:12:591
3558788995324,13cyclictest3636559-21ssh23:39:350
3558788995323,14cyclictest3574865-21ssh21:30:300
3558788995323,14cyclictest3574865-21ssh21:30:300
355879799523,18cyclictest3565010-21cron21:10:002
3558797995224,6cyclictest30-21ksoftirqd/219:55:002
3558797995222,5cyclictest3630592-21sh23:26:502
3558797995217,9cyclictest30-21ksoftirqd/221:31:072
3558797995217,9cyclictest30-21ksoftirqd/221:31:072
3558797995217,4cyclictest3660551-21sh00:29:592
355878899525,29cyclictest131rcu_preempt00:33:460
3558788995224,13cyclictest3664621-21ssh00:38:300
3558788995224,13cyclictest3574603-21ssh21:30:020
355878899521,36cyclictest3565748-21ssh21:11:180
3558802995113,6cyclictest131rcu_preempt19:35:013
3558802995113,6cyclictest131rcu_preempt19:35:013
3558797995114,5cyclictest0-21swapper/200:15:022
3558793995138,10cyclictest3592901-21ssh22:08:141
3558793995112,9cyclictest23-21ksoftirqd/123:37:331
3558788995141,6cyclictest12-21ksoftirqd/022:46:580
3558788995127,9cyclictest3599841-21ssh22:22:330
3558788995114,34cyclictest12-21ksoftirqd/021:41:310
3558802995016,22cyclictest3558167-21kworker/3:019:25:013
3558797995042,5cyclictest3563179-21kworker/2:320:50:012
3558797995036,11cyclictest30-21ksoftirqd/219:50:022
355879799503,21cyclictest3467602-21systemd-journal20:05:012
355879399509,13cyclictest3618070-21ssh23:00:271
3558793995035,12cyclictest3610726-21ssh22:45:081
3558793995031,14cyclictest3467602-21systemd-journal21:05:021
3558793995028,17cyclictest3569082-21ssh21:18:351
3558793995013,7cyclictest23-21ksoftirqd/122:29:251
3558793995010,7cyclictest23-21ksoftirqd/120:00:011
3558788995019,9cyclictest12-21ksoftirqd/020:20:010
355880299493,5cyclictest131rcu_preempt22:00:003
355880299493,5cyclictest131rcu_preempt22:00:003
3558797994925,10cyclictest30-21ksoftirqd/220:30:022
3558793994935,11cyclictest3658770-21ssh00:25:551
3558793994930,14cyclictest3587349-21ssh21:56:291
3558793994930,14cyclictest3587349-21ssh21:56:291
3558788994925,9cyclictest3582047-21ssh21:45:150
355878899492,16cyclictest3564500-21munin-run21:00:010
3558788994913,22cyclictest131rcu_preempt00:27:510
355880299488,6cyclictest131rcu_preempt00:10:003
3558802994833,12cyclictest3467602-21systemd-journal22:45:023
3558802994830,13cyclictest3655773-21systemctl00:20:003
3558802994830,13cyclictest3655773-21systemctl00:20:003
3558802994830,13cyclictest275-21dbus-daemon20:30:013
3558802994828,14cyclictest3467602-21systemd-journal21:40:003
3558802994824,18cyclictest275-21dbus-daemon21:55:013
3558797994841,5cyclictest3549852-21kworker/2:123:48:122
3558797994840,5cyclictest3543973-21kworker/2:219:20:012
3558797994821,5cyclictest30-21ksoftirqd/221:38:542
3558793994832,12cyclictest3564784-21munin-run21:05:011
3558793994831,14cyclictest3637424-21ssh23:41:031
3558788994822,11cyclictest3570883-21ssh21:22:090
3558788994814,10cyclictest12-21ksoftirqd/020:50:010
3558797994721,10cyclictest30-21ksoftirqd/219:45:012
3558793994734,10cyclictest3467602-21systemd-journal20:45:021
3558788994737,7cyclictest12-21ksoftirqd/019:45:010
3558788994712,6cyclictest131rcu_preempt22:00:020
3558788994712,6cyclictest131rcu_preempt22:00:020
3558802994630,9cyclictest3467602-21systemd-journal20:35:013
3558802994630,12cyclictest1-21systemd00:00:013
3558802994629,12cyclictest3467602-21systemd-journal22:15:003
3558802994624,11cyclictest37-21ksoftirqd/319:40:003
3558793994638,4cyclictest3558998-21munin-run19:15:011
3558788994611,12cyclictest12-21ksoftirqd/000:13:290
355880299451,12cyclictest131rcu_preempt23:50:003
3558793994530,11cyclictest3560787-21sshd19:47:031
355878899457,14cyclictest12-21ksoftirqd/019:25:000
3558788994538,4cyclictest16950irq/224-800000019:35:030
355878899453,6cyclictest131rcu_preempt20:25:020
3558788994531,9cyclictest151rcuc/019:35:010
3558788994531,9cyclictest151rcuc/019:35:010
3558802994432,9cyclictest3467602-21systemd-journal22:15:023
3558793994432,9cyclictest3559990-21sshd19:30:031
3558793994432,9cyclictest3559990-21sshd19:30:031
3558793994431,10cyclictest3563165-21munin-run20:35:011
3558793994431,10cyclictest3561566-21sshd20:00:241
355878899444,28cyclictest3558791-21latency_hist19:10:010
3558788994426,14cyclictest3613508-21ssh22:50:590
355880299435,6cyclictest131rcu_preempt22:02:033
355880299434,4cyclictest131rcu_preempt00:13:313
3558802994332,9cyclictest37-21ksoftirqd/322:52:083
3558802994330,9cyclictest3467602-21systemd-journal22:21:323
3558802994325,13cyclictest3467602-21systemd-journal23:05:013
3558793994330,9cyclictest3562408-21munin-run20:20:021
3558793994327,13cyclictest3562870-21sshd20:26:031
355878899434,6cyclictest131rcu_preempt20:05:020
355880299424,4cyclictest131rcu_preempt20:40:333
3558802994227,11cyclictest3467602-21systemd-journal22:40:013
3558802994225,13cyclictest1-21systemd21:00:013
355880299421,8cyclictest0-21swapper/323:26:093
355880299421,5cyclictest131rcu_preempt23:40:003
355880299421,5cyclictest131rcu_preempt22:35:013
355880299421,5cyclictest0-21swapper/320:37:033
3558797994222,6cyclictest30-21ksoftirqd/219:35:022
3558797994222,6cyclictest30-21ksoftirqd/219:35:022
3558793994230,7cyclictest309-21in:imuxsock19:45:001
3558793994228,10cyclictest3561034-21sshd19:50:221
3558788994215,21cyclictest131rcu_preempt20:35:010
355880299419,10cyclictest37-21ksoftirqd/323:25:023
355880299419,10cyclictest37-21ksoftirqd/323:25:013
355880299414,5cyclictest131rcu_preempt00:40:003
355880299413,6cyclictest131rcu_preempt21:32:043
355880299413,6cyclictest131rcu_preempt21:32:043
3558802994128,10cyclictest37-21ksoftirqd/321:14:323
3558802994125,4cyclictest37-21ksoftirqd/322:55:423
3558793994127,9cyclictest3559490-21sshd19:20:031
3558793994126,12cyclictest3559807-21cat19:30:011
355878899416,11cyclictest12-21ksoftirqd/019:15:010
355880299405,5cyclictest131rcu_preempt22:27:313
3558802994030,7cyclictest37-21ksoftirqd/323:15:303
3558802994026,10cyclictest3646585-21sshd00:00:123
3558797994031,5cyclictest3564768-21modprobe21:05:002
3558793994026,11cyclictest3467602-21systemd-journal20:10:581
3558793994026,10cyclictest48-21kcompactd020:51:301
3558788994030,7cyclictest3559966-21latency_hist19:30:020
3558788994025,10cyclictest12-21ksoftirqd/020:42:030
355880299394,6cyclictest131rcu_preempt21:30:023
355880299393,4cyclictest131rcu_preempt23:06:323
3558802993928,9cyclictest37-21ksoftirqd/322:06:413
3558802993928,9cyclictest37-21ksoftirqd/319:50:023
355880299392,5cyclictest131rcu_preempt21:05:013
3558802993925,10cyclictest37-21ksoftirqd/322:41:543
3558802993917,18cyclictest37-21ksoftirqd/300:26:203
355880299391,3cyclictest131rcu_preempt23:31:483
3558793993929,6cyclictest3560284-21sendmail19:40:001
3558788993910,9cyclictest131rcu_preempt21:05:010
355880299388,5cyclictest131rcu_preempt19:50:043
3558802993830,5cyclictest3581706-21latency_hist21:45:023
3558802993826,9cyclictest3561297-21sshd19:58:033
3558802993821,4cyclictest37-21ksoftirqd/300:23:003
3558802993821,4cyclictest37-21ksoftirqd/300:23:003
3558802993814,21cyclictest131rcu_preempt00:30:273
3558788993825,10cyclictest3562088-21sshd20:14:030
3558802993728,6cyclictest3564232-21sh20:55:003
3558802993728,6cyclictest3561585-21cron20:05:013
3558802993724,11cyclictest37-21ksoftirqd/319:45:013
3558802993627,6cyclictest3559233-21cron19:20:003
3558802993623,9cyclictest37-21ksoftirqd/320:50:023
3558802993620,12cyclictest3565006-21sshd21:09:043
3558797993628,5cyclictest30-21ksoftirqd/219:10:032
3558793993617,14cyclictest275-21dbus-daemon21:00:011
3558788993625,8cyclictest12-21ksoftirqd/020:30:020
3558788993625,8cyclictest12-21ksoftirqd/020:10:000
3558788993623,10cyclictest12-21ksoftirqd/019:50:010
355880299354,5cyclictest131rcu_preempt23:42:483
3558797993525,7cyclictest30-21ksoftirqd/219:15:022
3558802993422,9cyclictest37-21ksoftirqd/319:30:013
3558802993422,9cyclictest3562491-21wc20:20:023
3558802993420,10cyclictest275-21dbus-daemon20:25:003
3558802993415,14cyclictest275-21dbus-daemon19:15:013
3558793993417,13cyclictest3558869-21latency_hist19:10:021
3558802993320,10cyclictest3558809-21sed19:10:023
3558802993022,5cyclictest37-21ksoftirqd/320:14:033
355880299301,4cyclictest131rcu_preempt20:05:353
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional