You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-11 - 23:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Aug 11, 2026 12:45:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
39370229910214,5cyclictest131rcu_preempt07:55:013
393700899752,29cyclictest131rcu_preempt11:35:370
393700899742,45cyclictest12-21ksoftirqd/012:20:010
3937008997410,25cyclictest131rcu_preempt11:57:270
3937008997010,30cyclictest131rcu_preempt09:27:390
393700899672,7cyclictest12-21ksoftirqd/010:51:450
3937008996716,5cyclictest131rcu_preempt11:45:010
3937018996630,15cyclictest30-21ksoftirqd/211:45:002
393701899653,5cyclictest131rcu_preempt11:40:032
3937008996426,24cyclictest12-21ksoftirqd/011:47:100
393700899631,20cyclictest3937994-21modprobe07:30:000
3937022996218,6cyclictest37-21ksoftirqd/312:38:463
3937022996218,11cyclictest37-21ksoftirqd/309:15:013
393700899624,17cyclictest12-21ksoftirqd/009:20:000
3937008996220,19cyclictest16850irq/134-800000011:25:010
393700899614,5cyclictest131rcu_preempt10:35:010
3937008996133,11cyclictest4042221-21ssh12:37:180
393700899611,11cyclictest12-21ksoftirqd/009:34:170
393702299606,5cyclictest131rcu_preempt09:10:013
393701499602,12cyclictest131rcu_preempt11:40:021
393700899606,11cyclictest12-21ksoftirqd/010:25:020
393700899606,11cyclictest12-21ksoftirqd/010:25:010
393700899592,9cyclictest12-21ksoftirqd/008:20:010
3937008995925,30cyclictest151rcuc/010:00:010
3937008995912,19cyclictest12-21ksoftirqd/009:25:010
393702299581,9cyclictest3993724-21ssh10:55:233
3937018995842,7cyclictest3942944-21kworker/2:110:17:372
393700899584,17cyclictest275-21dbus-daemon11:00:000
3937008995820,18cyclictest16850irq/134-800000011:15:010
3937008995820,18cyclictest16850irq/134-800000011:15:000
3937018995744,5cyclictest3942944-21kworker/2:110:14:482
3937014995734,14cyclictest3467602-21systemd-journal09:45:001
393700899577,9cyclictest12-21ksoftirqd/012:15:010
393700899577,38cyclictest12-21ksoftirqd/012:07:380
393700899576,6cyclictest131rcu_preempt11:19:180
3937008995746,7cyclictest16850irq/134-800000011:10:010
3937008995730,12cyclictest3980988-21ssh10:29:040
3937008995729,13cyclictest3984690-21ssh10:36:370
3937008995726,15cyclictest3961641-21ssh09:48:270
3937008995714,5cyclictest131rcu_preempt10:15:360
393702299561,8cyclictest3969524-21latency_hist10:05:013
393700899565,23cyclictest131rcu_preempt12:35:020
3937008995610,6cyclictest131rcu_preempt09:40:130
3937022995523,7cyclictest37-21ksoftirqd/308:20:003
3937022995514,9cyclictest131rcu_preempt11:07:343
3937022995513,6cyclictest131rcu_preempt11:45:003
3937022995511,6cyclictest131rcu_preempt11:38:443
3937018995547,5cyclictest3942944-21kworker/2:109:44:002
3937018995546,6cyclictest3952763-21kworker/2:211:51:552
3937018995545,6cyclictest3952763-21kworker/2:211:08:162
3937018995541,6cyclictest3942944-21kworker/2:110:33:392
393700899555,5cyclictest131rcu_preempt11:54:020
393700899551,9cyclictest12-21ksoftirqd/007:25:010
393700899551,39cyclictest4036555-21ssh12:25:210
3937008995512,11cyclictest12-21ksoftirqd/009:40:000
3937022995426,19cyclictest37-21ksoftirqd/309:32:063
393702299541,8cyclictest0-21swapper/312:03:473
3937018995445,6cyclictest3942944-21kworker/2:112:38:552
3937018995445,6cyclictest3942944-21kworker/2:112:03:552
393700899548,8cyclictest12-21ksoftirqd/007:45:000
393700899545,10cyclictest12-21ksoftirqd/009:00:000
3937008995426,13cyclictest3973897-21ssh10:14:200
393700899542,37cyclictest3963490-21ssh09:52:190
3937008995416,5cyclictest131rcu_preempt10:00:530
393702299533,11cyclictest37-21ksoftirqd/309:52:343
3937022995317,7cyclictest37-21ksoftirqd/309:30:023
3937022995314,10cyclictest37-21ksoftirqd/311:25:023
3937022995313,5cyclictest131rcu_preempt07:35:003
3937022995311,6cyclictest131rcu_preempt11:10:023
3937022995311,6cyclictest131rcu_preempt11:10:023
3937018995346,4cyclictest3942944-21kworker/2:109:45:402
3937018995345,5cyclictest3952763-21kworker/2:211:34:562
3937018995345,5cyclictest3942944-21kworker/2:112:15:392
3937008995337,12cyclictest275-21dbus-daemon11:05:020
3937008995328,19cyclictest275-21dbus-daemon10:50:020
393702299529,8cyclictest131rcu_preempt11:52:333
393702299529,6cyclictest131rcu_preempt08:39:343
393702299529,5cyclictest131rcu_preempt10:55:013
393702299528,6cyclictest131rcu_preempt10:15:283
393702299528,5cyclictest131rcu_preempt10:14:533
3937022995231,8cyclictest37-21ksoftirqd/312:22:143
3937022995224,7cyclictest37-21ksoftirqd/308:25:013
3937022995218,6cyclictest37-21ksoftirqd/312:14:093
393702299521,5cyclictest131rcu_preempt09:00:003
3937022995213,6cyclictest131rcu_preempt12:30:023
3937018995244,5cyclictest3942944-21kworker/2:112:20:492
3937018995244,5cyclictest3942944-21kworker/2:111:45:472
3937018995244,5cyclictest3942944-21kworker/2:110:49:422
3937018995244,5cyclictest3942944-21kworker/2:110:00:152
3937018995244,5cyclictest3942944-21kworker/2:109:38:122
3937018995244,5cyclictest3942944-21kworker/2:109:20:492
3937018995244,5cyclictest3925386-21kworker/2:009:14:332
3937018995243,5cyclictest3942944-21kworker/2:110:08:072
393700899526,7cyclictest12-21ksoftirqd/011:33:220
393700899525,9cyclictest12-21ksoftirqd/007:31:320
3937008995226,11cyclictest3943994-21ssh09:11:190
3937008995219,30cyclictest12-21ksoftirqd/010:40:100
3937008995211,8cyclictest12-21ksoftirqd/012:05:000
393702299519,6cyclictest131rcu_preempt07:45:003
393702299518,6cyclictest131rcu_preempt10:40:023
3937022995110,5cyclictest131rcu_preempt11:30:233
3937022995110,5cyclictest131rcu_preempt11:18:463
3937018995143,5cyclictest3952763-21kworker/2:211:26:152
3937018995143,5cyclictest3942944-21kworker/2:111:04:042
3937018995143,5cyclictest3942944-21kworker/2:110:35:582
3937018995143,5cyclictest3942944-21kworker/2:110:27:322
3937018995143,5cyclictest3942944-21kworker/2:109:29:022
3937018995142,6cyclictest3942944-21kworker/2:110:57:482
3937018995141,7cyclictest3925386-21kworker/2:009:15:062
3937008995122,15cyclictest3970818-21ssh10:07:450
3937008995120,5cyclictest131rcu_preempt12:21:000
3937008995112,6cyclictest16850irq/134-800000011:29:160
393702299506,8cyclictest37-21ksoftirqd/310:24:093
393702299506,8cyclictest37-21ksoftirqd/310:24:083
393702299502,5cyclictest131rcu_preempt10:35:013
3937022995012,6cyclictest131rcu_preempt09:40:033
3937022995010,6cyclictest131rcu_preempt12:00:013
3937018995042,5cyclictest3942944-21kworker/2:112:33:582
3937018995042,5cyclictest3942944-21kworker/2:112:25:292
3937018995042,5cyclictest3942944-21kworker/2:112:10:092
3937018995042,5cyclictest3942944-21kworker/2:111:22:032
3937018995042,5cyclictest3942944-21kworker/2:111:17:052
3937018995042,5cyclictest3942944-21kworker/2:111:10:062
3937018995042,5cyclictest3942944-21kworker/2:111:10:062
3937018995042,5cyclictest3942944-21kworker/2:110:52:282
3937018995042,5cyclictest3942944-21kworker/2:109:50:462
3937018995041,5cyclictest3942944-21kworker/2:109:32:362
3937018995040,6cyclictest3942944-21kworker/2:112:07:402
3937018995040,6cyclictest3942944-21kworker/2:110:40:512
3937018995036,6cyclictest3925386-21kworker/2:008:18:192
393701499502,35cyclictest221rcuc/109:50:021
393700899503,39cyclictest3941616-21modprobe08:40:000
393700899502,43cyclictest3467602-21systemd-journal07:40:010
393700899502,43cyclictest3467602-21systemd-journal07:40:010
393700899502,17cyclictest1-21systemd08:00:010
393702299497,19cyclictest131rcu_preempt08:15:023
393702299493,4cyclictest131rcu_preempt10:00:003
393702299492,5cyclictest131rcu_preempt12:10:013
3937022994913,9cyclictest131rcu_preempt09:47:073
3937022994912,9cyclictest131rcu_preempt09:15:043
3937018994941,5cyclictest3952763-21kworker/2:211:55:032
3937018994941,5cyclictest3942944-21kworker/2:110:20:082
3937018994941,5cyclictest3942944-21kworker/2:110:20:072
393700899493,38cyclictest3943214-21unin-run09:10:010
3937008994933,11cyclictest3467602-21systemd-journal08:10:010
3937008994913,6cyclictest131rcu_preempt08:05:020
3937008994913,6cyclictest131rcu_preempt08:05:020
393702299487,7cyclictest37-21ksoftirqd/310:45:003
393702299487,5cyclictest131rcu_preempt10:27:043
393702299487,11cyclictest37-21ksoftirqd/309:25:003
393702299486,5cyclictest131rcu_preempt11:47:113
3937022994820,15cyclictest37-21ksoftirqd/309:37:283
3937022994816,6cyclictest131rcu_preempt09:05:023
393702299481,5cyclictest131rcu_preempt07:20:013
393702299481,5cyclictest0-21swapper/310:49:453
3937022994814,5cyclictest131rcu_preempt12:35:003
3937022994811,8cyclictest37-21ksoftirqd/312:15:383
3937018994841,4cyclictest3942944-21kworker/2:109:57:322
3937018994840,5cyclictest3925386-21kworker/2:008:21:472
3937018994840,5cyclictest3925386-21kworker/2:007:42:572
3937018994832,12cyclictest275-21dbus-daemon09:00:012
393701499484,3cyclictest131rcu_preempt07:55:011
393701499483,4cyclictest131rcu_preempt12:24:391
3937014994832,11cyclictest3947990-21cron09:20:001
393701499482,23cyclictest23-21ksoftirqd/111:25:021
393702299479,4cyclictest131rcu_preempt08:45:013
393702299474,6cyclictest131rcu_preempt11:30:013
3937022994713,5cyclictest131rcu_preempt10:10:003
3937022994711,6cyclictest131rcu_preempt11:00:353
3937018994739,5cyclictest3925386-21kworker/2:009:02:352
3937018994739,5cyclictest3925386-21kworker/2:007:55:002
3937018994739,5cyclictest3925386-21kworker/2:007:22:182
393701499472,3cyclictest131rcu_preempt10:44:191
393702299468,6cyclictest131rcu_preempt08:35:023
393702299462,6cyclictest131rcu_preempt08:05:013
393702299462,6cyclictest131rcu_preempt08:05:013
393702299461,9cyclictest0-21swapper/307:21:323
393702299461,6cyclictest3942426-21latency_hist08:55:013
3937018994638,5cyclictest3925386-21kworker/2:008:50:382
3937018994638,5cyclictest3925386-21kworker/2:008:10:152
3937018994638,5cyclictest3925386-21kworker/2:007:38:262
3937018994638,5cyclictest3925386-21kworker/2:007:38:262
3937018994638,5cyclictest3925386-21kworker/2:007:27:032
393701499463,3cyclictest131rcu_preempt11:16:481
393701499461,7cyclictest131rcu_preempt07:35:001
393700899464,11cyclictest12-21ksoftirqd/008:35:010
393700899463,7cyclictest12-21ksoftirqd/008:30:000
3937008994625,15cyclictest275-21dbus-daemon08:55:010
3937008994611,5cyclictest131rcu_preempt08:25:010
393702299459,5cyclictest131rcu_preempt07:40:023
393702299459,5cyclictest131rcu_preempt07:40:023
393702299458,5cyclictest131rcu_preempt07:50:003
393702299458,5cyclictest131rcu_preempt07:50:003
393702299455,4cyclictest131rcu_preempt07:15:013
3937022994511,5cyclictest131rcu_preempt08:50:013
3937018994538,4cyclictest3925386-21kworker/2:008:04:022
3937018994538,4cyclictest3925386-21kworker/2:008:04:022
3937018994538,4cyclictest3925386-21kworker/2:007:18:532
3937018994537,5cyclictest3943305-21latency_hist09:10:022
3937018994537,5cyclictest3925386-21kworker/2:008:30:312
393701499452,3cyclictest131rcu_preempt10:13:161
393701499451,40cyclictest23-21ksoftirqd/109:59:591
3937014994513,16cyclictest4019588-21ssh11:50:021
393700899452,11cyclictest12-21ksoftirqd/007:10:030
393700899451,7cyclictest3942932-21munin-run09:05:000
393702299444,9cyclictest37-21ksoftirqd/307:10:023
3937022994433,9cyclictest37-21ksoftirqd/308:00:013
3937022994412,4cyclictest131rcu_preempt08:30:003
3937018994437,4cyclictest3925386-21kworker/2:008:48:282
3937018994437,4cyclictest3925386-21kworker/2:008:29:392
3937018994437,4cyclictest3925386-21kworker/2:007:55:092
3937014994432,9cyclictest1-21systemd09:35:001
3937014994430,11cyclictest1-21systemd08:05:011
3937014994430,11cyclictest1-21systemd08:05:011
3937014994428,12cyclictest23-21ksoftirqd/111:10:291
3937014994428,12cyclictest23-21ksoftirqd/111:10:291
393701499441,3cyclictest131rcu_preempt11:55:061
393700899446,9cyclictest12-21ksoftirqd/008:50:010
393700899446,7cyclictest12-21ksoftirqd/007:15:010
393702299436,6cyclictest131rcu_preempt07:30:013
393702299433,6cyclictest131rcu_preempt08:10:003
3937018994337,4cyclictest3925386-21kworker/2:008:05:462
3937018994335,5cyclictest3925386-21kworker/2:008:37:532
3937018994335,5cyclictest3925386-21kworker/2:007:10:052
3937018994326,11cyclictest3938214-21kworker/u8:007:35:022
3937014994332,8cyclictest23-21ksoftirqd/110:30:021
3937014994330,10cyclictest3467602-21systemd-journal07:48:321
3937014994330,10cyclictest3467602-21systemd-journal07:48:321
3937014994329,10cyclictest23-21ksoftirqd/112:29:171
3937014994325,13cyclictest275-21dbus-daemon10:05:011
393700899436,8cyclictest12-21ksoftirqd/007:19:590
3937008994311,6cyclictest131rcu_preempt08:13:340
3937018994235,4cyclictest3925386-21kworker/2:008:43:152
393701499427,6cyclictest131rcu_preempt10:09:181
3937014994231,7cyclictest3995702-21sh10:59:591
3937014994131,8cyclictest23-21ksoftirqd/107:25:021
3937014994131,7cyclictest23-21ksoftirqd/111:31:471
3937014994131,7cyclictest23-21ksoftirqd/108:15:031
3937014994130,8cyclictest3467602-21systemd-journal12:14:011
3937014994123,12cyclictest275-21dbus-daemon09:00:001
393701499411,3cyclictest131rcu_preempt09:22:231
393701499411,3cyclictest131rcu_preempt07:40:021
3937008994122,14cyclictest275-21dbus-daemon07:55:000
3937018994032,5cyclictest3925386-21kworker/2:007:48:382
3937018994032,5cyclictest3925386-21kworker/2:007:48:382
3937014994031,6cyclictest23-21ksoftirqd/108:30:011
3937014994030,7cyclictest4029170-21date12:10:011
3937014994027,9cyclictest309-21in:imuxsock08:00:001
3937014994024,11cyclictest1-21systemd07:15:011
393701499402,3cyclictest131rcu_preempt10:20:021
3937014994023,12cyclictest1-21systemd09:15:001
393701499402,14cyclictest4041095-21grep12:35:031
393701499401,3cyclictest131rcu_preempt11:23:051
3937008994011,5cyclictest131rcu_preempt08:40:020
3937014993931,5cyclictest23-21ksoftirqd/112:36:161
3937014993929,8cyclictest23-21ksoftirqd/111:52:091
3937014993929,5cyclictest309-21in:imuxsock12:05:011
3937014993928,8cyclictest23-21ksoftirqd/109:10:031
3937014993927,9cyclictest23-21ksoftirqd/110:38:281
3937014993927,9cyclictest23-21ksoftirqd/110:32:531
3937014993927,10cyclictest23-21ksoftirqd/110:48:321
3937014993926,11cyclictest23-21ksoftirqd/109:51:431
3937014993924,10cyclictest23-21ksoftirqd/109:05:001
393701499392,3cyclictest131rcu_preempt11:45:011
393701499391,3cyclictest131rcu_preempt11:09:001
393701499391,12cyclictest3951153-21ssh09:26:161
393701499385,3cyclictest131rcu_preempt08:20:001
393701499383,3cyclictest131rcu_preempt09:36:311
3937014993827,7cyclictest23-21ksoftirqd/108:40:481
3937014993826,9cyclictest23-21ksoftirqd/111:05:011
3937014993825,9cyclictest309-21in:imuxsock07:30:011
3937014993821,12cyclictest23-21ksoftirqd/110:25:011
3937014993821,12cyclictest23-21ksoftirqd/110:25:011
3937014993820,12cyclictest1-21systemd08:55:021
3937008993828,7cyclictest12-21ksoftirqd/007:50:010
3937008993828,7cyclictest12-21ksoftirqd/007:50:000
393701499372,5cyclictest131rcu_preempt10:55:021
3937014993725,8cyclictest3941349-21sshd08:34:341
3937014993721,11cyclictest275-21dbus-daemon08:50:001
393701499371,3cyclictest131rcu_preempt07:40:001
393701499371,3cyclictest131rcu_preempt07:40:001
3937014993628,5cyclictest3941722-21latency_hist08:40:031
3937014993623,9cyclictest309-21in:imuxsock07:20:001
393701499352,7cyclictest4032684-21ssh12:17:261
393701499342,3cyclictest131rcu_preempt07:10:031
3937014993421,10cyclictest23-21ksoftirqd/108:10:011
393701499323,5cyclictest131rcu_preempt08:23:341
393701899291,5cyclictest131rcu_preempt07:10:042
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional