You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-01 - 17:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri May 01, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
484727999622,5cyclictest16850irq/134-800000011:05:010
484727998626,13cyclictest511079-21ssh09:56:220
484727998527,26cyclictest541659-21ssh11:07:050
484727998428,11cyclictest546254-21ssh11:17:470
484727998417,56cyclictest16850irq/134-800000009:51:110
484727998229,12cyclictest514479-21ssh10:04:260
484727998228,12cyclictest538251-21ssh10:59:270
484727998228,12cyclictest538251-21ssh10:59:270
48472799822,40cyclictest495252-21sh09:20:000
48472799812,40cyclictest517878-21ssh10:12:050
48472799781,39cyclictest575878-21ssh12:26:090
48472799771,39cyclictest562077-21ssh11:54:360
484727997712,6cyclictest16850irq/134-800000009:39:530
484727997625,15cyclictest501020-21ssh09:33:100
484734997518,34cyclictest487019-21kworker/u8:207:55:022
484727997537,4cyclictest16950irq/224-800000010:45:020
484727997522,10cyclictest1-21systemd09:10:010
484727997435,4cyclictest16950irq/224-800000012:05:000
484727997434,4cyclictest16950irq/224-800000010:35:090
484727997414,17cyclictest16850irq/134-800000011:23:560
48472799741,33cyclictest487797-21/usr/sbin/munin08:10:010
484727997326,14cyclictest521973-21ssh10:21:190
484727997318,6cyclictest16850irq/134-800000012:20:220
484727997316,19cyclictest16850irq/134-800000009:49:590
484727997313,16cyclictest16850irq/134-800000011:25:180
484727997313,16cyclictest16850irq/134-800000011:25:170
484734997230,38cyclictest3467602-21systemd-journal10:30:322
484734997226,40cyclictest275-21dbus-daemon09:50:012
484727997235,4cyclictest16950irq/224-800000012:35:020
484727997218,17cyclictest16850irq/134-800000011:55:440
484727997218,17cyclictest16850irq/134-800000011:55:440
48472799721,58cyclictest534219-21sh10:50:010
484727997116,18cyclictest16850irq/134-800000012:09:430
484727996827,13cyclictest552270-21ssh11:31:320
484734996721,13cyclictest30-21ksoftirqd/212:36:202
48472799675,21cyclictest16850irq/134-800000011:15:020
484734996617,10cyclictest0-21swapper/212:35:022
484730996640,18cyclictest275-21dbus-daemon09:10:011
484727996621,11cyclictest571807-21ssh12:16:510
484734996532,28cyclictest30-21ksoftirqd/211:10:012
484727996517,16cyclictest16850irq/134-800000011:41:350
484734996421,5cyclictest490578-21latency_hist09:00:022
484734996418,10cyclictest566513-21sshd12:04:542
48472799645,34cyclictest16850irq/134-800000012:37:020
48472799645,18cyclictest52407450ssh10:26:180
484727996417,40cyclictest3467602-21systemd-journal08:20:010
484734996326,5cyclictest274-21cron12:00:002
484734996326,5cyclictest274-21cron12:00:002
484734996325,16cyclictest275-21dbus-daemon10:15:012
484727996314,16cyclictest516527-21ssh10:09:160
484734996227,32cyclictest567086-21sh12:05:442
484734996125,5cyclictest499854-21sh09:30:152
484734996123,9cyclictest30-21ksoftirqd/208:25:002
484734996123,4cyclictest519664-21sh10:16:032
484734996123,10cyclictest30-21ksoftirqd/211:01:392
484727996111,18cyclictest16850irq/134-800000009:21:540
484727996111,17cyclictest16850irq/134-800000009:41:300
484734996027,28cyclictest1-21systemd09:15:012
484734996024,5cyclictest544897-21sh11:14:492
484734996022,5cyclictest545489-21sh11:15:512
484734996020,11cyclictest488113-21sort08:15:002
484727996030,13cyclictest559260-21ssh11:47:580
484734995918,9cyclictest30-21ksoftirqd/211:33:592
484727995929,14cyclictest519966-21ssh10:16:530
484734995850,5cyclictest538540-21kworker/2:112:10:572
484734995825,6cyclictest30-21ksoftirqd/212:17:352
484734995822,11cyclictest30-21ksoftirqd/212:27:322
484727995828,13cyclictest569374-21ssh12:11:090
484734995722,4cyclictest503868-21latency_hist09:40:022
484734995718,8cyclictest30-21ksoftirqd/210:20:022
484734995717,10cyclictest530282-1kworker/u9:210:44:412
484734995619,9cyclictest30-21ksoftirqd/209:27:592
484734995525,7cyclictest30-21ksoftirqd/211:29:142
484734995525,7cyclictest30-21ksoftirqd/211:29:142
48473499552,22cyclictest3467602-21systemd-journal09:55:012
484734995517,9cyclictest530282-1kworker/u9:210:45:102
484730995531,19cyclictest275-21dbus-daemon11:55:001
484734995429,22cyclictest30-21ksoftirqd/208:35:022
484734995429,22cyclictest30-21ksoftirqd/208:35:022
484734995417,10cyclictest30-21ksoftirqd/210:09:422
484730995437,13cyclictest555770-21cron11:40:001
484730995437,13cyclictest555770-21cron11:40:001
484730995433,15cyclictest1-21systemd07:35:011
484727995424,26cyclictest534439-21ssh10:50:060
484734995324,4cyclictest512508-21latency_hist10:00:022
484734995320,10cyclictest489561-21cat08:40:022
484730995324,20cyclictest275-21dbus-daemon10:05:011
484727995318,31cyclictest16850irq/134-800000009:13:270
484727995318,30cyclictest16850irq/134-800000009:27:320
484734995232,15cyclictest3467602-21systemd-journal07:35:002
484734995225,7cyclictest30-21ksoftirqd/210:50:362
484734995216,10cyclictest30-21ksoftirqd/211:22:462
484730995237,10cyclictest488318-21systemctl08:20:011
484730995231,12cyclictest485192-21sendmail07:20:001
484740995139,8cyclictest490392-21run-parts09:00:003
484740995120,17cyclictest37-21ksoftirqd/308:30:023
484734995143,5cyclictest488350-21kworker/2:310:58:112
484734995143,5cyclictest488350-21kworker/2:310:58:112
484734995123,9cyclictest30-21ksoftirqd/207:20:032
484734995123,9cyclictest30-21ksoftirqd/207:20:022
484734995121,11cyclictest30-21ksoftirqd/208:45:012
484730995129,13cyclictest23-21ksoftirqd/111:05:121
484727995117,16cyclictest16850irq/134-800000010:34:330
484727995116,17cyclictest16850irq/134-800000011:36:550
484727995116,17cyclictest16850irq/134-800000011:36:550
484740995012,5cyclictest131rcu_preempt10:50:503
48473499502,15cyclictest525084-21ssh10:28:512
484730995033,11cyclictest1-21systemd11:35:001
484730995029,16cyclictest275-21dbus-daemon12:25:011
484730995021,23cyclictest275-21dbus-daemon09:00:001
48472799507,25cyclictest490099-21kworker/u8:209:00:020
484734994942,4cyclictest488350-21kworker/2:309:22:182
484734994922,4cyclictest574788-21sshd12:23:542
484730994930,14cyclictest3467602-21systemd-journal08:30:011
484730994929,14cyclictest275-21dbus-daemon10:30:011
484730994928,14cyclictest3467602-21systemd-journal08:25:001
484740994834,11cyclictest1-21systemd11:10:013
484734994840,5cyclictest488350-21kworker/2:311:51:172
484734994839,6cyclictest488350-21kworker/2:309:05:012
484734994820,7cyclictest30-21ksoftirqd/209:19:132
484730994824,18cyclictest275-21dbus-daemon09:35:011
484730994823,19cyclictest275-21dbus-daemon10:00:011
484740994718,19cyclictest37-21ksoftirqd/310:30:023
484734994740,4cyclictest487828-21kworker/2:010:01:542
484734994739,5cyclictest519418-21kworker/2:210:35:442
484734994739,5cyclictest487828-21kworker/2:009:44:492
484734994739,5cyclictest487828-21kworker/2:008:30:012
484734994737,6cyclictest488350-21kworker/2:311:46:492
484730994736,8cyclictest23-21ksoftirqd/108:55:011
484730994733,8cyclictest484738-21wc07:10:021
484730994733,8cyclictest484738-21wc07:10:011
484730994732,12cyclictest3467602-21systemd-journal11:50:001
484730994732,11cyclictest275-21dbus-daemon09:40:011
48473099472,5cyclictest131rcu_preempt12:20:041
484727994719,23cyclictest16850irq/134-800000007:59:510
48474099464,3cyclictest131rcu_preempt09:47:153
48474099463,3cyclictest131rcu_preempt09:41:193
484730994634,8cyclictest23-21ksoftirqd/109:15:021
484730994628,14cyclictest551564-21cut11:30:021
484730994628,14cyclictest551564-21cut11:30:021
48473099461,3cyclictest131rcu_preempt10:19:521
484730994610,21cyclictest486224-21modprobe07:40:011
484730994610,21cyclictest486224-21modprobe07:40:001
484727994630,12cyclictest486240-21systemctl07:40:010
484727994630,12cyclictest486240-21systemctl07:40:000
48474099452,3cyclictest131rcu_preempt08:25:003
484734994537,5cyclictest488350-21kworker/2:311:44:242
484734994537,5cyclictest484337-21kworker/2:007:10:022
484734994537,5cyclictest484337-21kworker/2:007:10:022
48473099458,9cyclictest23-21ksoftirqd/110:15:011
48473099455,28cyclictest547132-21modprobe11:20:001
48473099454,29cyclictest581640-21ssh12:40:011
484730994521,17cyclictest275-21dbus-daemon09:55:011
48472799453,4cyclictest131rcu_preempt07:15:000
48474099448,3cyclictest131rcu_preempt09:36:233
484740994433,8cyclictest486781-21cat07:50:013
484740994432,8cyclictest1-21systemd11:30:013
484740994432,8cyclictest1-21systemd11:30:013
484740994431,10cyclictest3467602-21systemd-journal11:56:023
484740994431,10cyclictest3467602-21systemd-journal11:56:023
484740994429,10cyclictest3467602-21systemd-journal09:10:003
484740994426,13cyclictest275-21dbus-daemon08:40:003
48474099441,3cyclictest131rcu_preempt11:52:223
484740994410,5cyclictest131rcu_preempt10:01:413
484730994430,10cyclictest23-21ksoftirqd/108:40:021
484730994429,11cyclictest1-21systemd08:50:011
484730994427,12cyclictest3467602-21systemd-journal11:00:001
484730994427,12cyclictest3467602-21systemd-journal11:00:001
484730994426,13cyclictest23-21ksoftirqd/110:35:021
484730994421,6cyclictest23-21ksoftirqd/110:45:011
48472799443,8cyclictest485191-21latency_hist07:19:590
484727994432,9cyclictest1-21systemd07:30:010
48474099434,3cyclictest131rcu_preempt12:39:253
48474099433,3cyclictest131rcu_preempt11:43:493
484740994331,9cyclictest37-21ksoftirqd/311:22:563
484740994328,12cyclictest487360-21sed08:00:023
48474099432,24cyclictest487019-21kworker/u8:208:10:033
48474099431,4cyclictest131rcu_preempt12:00:303
48474099431,37cyclictest1-21systemd11:20:013
484734994332,8cyclictest490105-21sshd08:52:522
484734994328,11cyclictest30-21ksoftirqd/211:37:292
484734994328,11cyclictest30-21ksoftirqd/211:37:292
48473099436,25cyclictest487254-21modprobe08:00:001
48473099435,27cyclictest309-21in:imuxsock07:25:011
48473099435,27cyclictest309-21in:imuxsock07:25:001
48473099433,15cyclictest564467-21cat12:00:011
48473099433,15cyclictest564467-21cat12:00:011
48473099432,4cyclictest131rcu_preempt11:25:021
48473099432,12cyclictest571067-21ssh12:15:021
48472799434,3cyclictest131rcu_preempt08:25:020
484727994329,10cyclictest485437-21sshd07:21:520
484727994329,10cyclictest485437-21sshd07:21:520
48474099424,3cyclictest131rcu_preempt09:12:453
48474099423,3cyclictest131rcu_preempt10:56:563
48474099423,3cyclictest131rcu_preempt10:56:563
484740994232,7cyclictest37-21ksoftirqd/311:15:023
484740994229,10cyclictest37-21ksoftirqd/311:50:013
484740994228,10cyclictest485188-21cron07:19:593
484740994228,10cyclictest3467602-21systemd-journal09:32:503
484740994226,12cyclictest275-21dbus-daemon09:25:013
48474099422,3cyclictest131rcu_preempt12:28:323
48474099422,3cyclictest131rcu_preempt10:06:043
48473499425,4cyclictest131rcu_preempt08:50:022
484734994228,11cyclictest487526-21sshd08:04:512
48473099424,3cyclictest131rcu_preempt12:26:251
484730994235,4cyclictest231ksoftirqd/108:14:341
484730994227,12cyclictest23-21ksoftirqd/110:36:151
484730994225,11cyclictest309-21in:imuxsock07:29:501
48473099422,3cyclictest131rcu_preempt09:17:401
484730994223,11cyclictest309-21in:imuxsock09:00:031
484730994219,4cyclictest23-21ksoftirqd/109:44:251
484727994229,9cyclictest3467602-21systemd-journal07:40:030
484727994227,12cyclictest12-21ksoftirqd/008:35:010
484727994227,12cyclictest12-21ksoftirqd/008:35:010
484727994227,12cyclictest12-21ksoftirqd/008:15:010
48472799421,39cyclictest0-21swapper/007:10:020
48472799421,39cyclictest0-21swapper/007:10:010
48474099413,3cyclictest131rcu_preempt11:33:193
48474099412,7cyclictest0-21swapper/310:32:053
484740994122,5cyclictest37-21ksoftirqd/312:20:023
484734994133,5cyclictest487828-21kworker/2:009:08:582
484734994133,5cyclictest486771-21kworker/2:108:10:012
48473499411,4cyclictest131rcu_preempt07:40:022
48473499411,4cyclictest131rcu_preempt07:40:022
48473099413,6cyclictest131rcu_preempt12:05:001
484730994128,9cyclictest523295-21sshd10:24:511
484727994134,4cyclictest16950irq/224-800000007:55:020
484740994031,6cyclictest37-21ksoftirqd/310:00:023
484740994029,8cyclictest37-21ksoftirqd/308:05:003
484740994025,11cyclictest3467602-21systemd-journal12:32:563
484740994024,12cyclictest3467602-21systemd-journal07:40:013
484740994024,12cyclictest3467602-21systemd-journal07:40:013
484740994011,5cyclictest131rcu_preempt10:49:163
484734994024,12cyclictest275-21dbus-daemon08:20:012
48473099406,23cyclictest3467602-21systemd-journal12:10:001
484730994030,6cyclictest497406-21latency_hist09:25:021
484730994028,9cyclictest23-21ksoftirqd/110:47:311
48473099402,34cyclictest48-21kcompactd012:30:241
484730994019,15cyclictest0-21swapper/107:45:011
484727994033,4cyclictest16950irq/224-800000008:49:230
484727994025,10cyclictest12-21ksoftirqd/007:35:010
484727994024,11cyclictest275-21dbus-daemon08:45:010
48474099395,7cyclictest131rcu_preempt10:15:033
48474099393,3cyclictest131rcu_preempt10:15:453
484740993927,7cyclictest488296-21cron08:20:003
484740993926,11cyclictest37-21ksoftirqd/311:40:023
484740993926,11cyclictest37-21ksoftirqd/311:40:023
48474099392,4cyclictest131rcu_preempt08:45:023
48474099392,3cyclictest131rcu_preempt10:35:263
484740993921,13cyclictest3467602-21systemd-journal09:05:013
484740993919,15cyclictest275-21dbus-daemon07:45:013
48474099391,3cyclictest131rcu_preempt12:24:143
484730993927,9cyclictest23-21ksoftirqd/108:10:001
484730993927,8cyclictest536356-21cron10:55:001
484730993923,12cyclictest23-21ksoftirqd/108:05:011
48473099391,3cyclictest131rcu_preempt11:00:101
484740993830,6cyclictest37-21ksoftirqd/312:10:333
484740993829,5cyclictest309-21in:imuxsock09:55:013
484740993826,10cyclictest37-21ksoftirqd/307:15:003
48474099382,3cyclictest131rcu_preempt09:17:073
484734993828,7cyclictest486475-21sshd07:41:502
484730993827,9cyclictest23-21ksoftirqd/111:42:321
484730993822,10cyclictest508346-21sh09:50:021
484730993821,13cyclictest23-21ksoftirqd/110:09:201
48472799383,31cyclictest0-21swapper/008:00:450
484727993824,10cyclictest489333-21sshd08:39:510
484740993725,9cyclictest37-21ksoftirqd/309:30:023
48474099371,3cyclictest131rcu_preempt12:09:173
484734993726,8cyclictest30-21ksoftirqd/207:17:242
484730993726,7cyclictest489223-21latency_hist08:35:021
484730993726,7cyclictest489223-21latency_hist08:35:021
484730993725,10cyclictest23-21ksoftirqd/109:29:331
484730993720,10cyclictest309-21in:imuxsock07:50:011
484730993714,16cyclictest275-21dbus-daemon07:55:001
484727993727,7cyclictest12-21ksoftirqd/008:55:000
484740993628,5cyclictest539214-21sh11:01:173
484740993624,8cyclictest490146-21unin-run08:55:013
484740993622,11cyclictest1-21systemd07:35:013
48473499364,9cyclictest30-21ksoftirqd/207:59:102
484734993625,8cyclictest30-21ksoftirqd/207:47:092
484740993523,9cyclictest37-21ksoftirqd/310:40:433
484740993521,10cyclictest0-21swapper/308:50:013
48473499352,3cyclictest131rcu_preempt07:30:012
484730993527,6cyclictest543315-21ssh11:10:461
48472799353,28cyclictest0-21swapper/009:02:030
484727993522,9cyclictest275-21dbus-daemon07:50:010
48474099343,4cyclictest131rcu_preempt10:24:393
48473099343,24cyclictest489650-21wc08:45:011
484727993425,5cyclictest19350irq/34-mmc108:25:230
48474099334,3cyclictest131rcu_preempt07:25:013
48474099334,3cyclictest131rcu_preempt07:25:013
484730993323,8cyclictest23-21ksoftirqd/107:15:021
484740993222,7cyclictest37-21ksoftirqd/308:15:003
48474099313,23cyclictest485724-21cut07:29:593
484740993122,7cyclictest37-21ksoftirqd/307:55:033
484740993115,13cyclictest489082-21sshd08:34:513
484740993115,13cyclictest489082-21sshd08:34:503
484734992922,4cyclictest484927-21munin-plugin-st07:14:592
48474099284,18cyclictest484816-21grep07:10:023
48474099284,18cyclictest484816-21grep07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional