You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-08 - 19:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue Sep 08, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
285911799129125,2cyclictest0-21swapper/009:05:030
2859117998616,14cyclictest2936958-21ssh11:41:390
2859117998616,14cyclictest2936958-21ssh11:41:390
2859117998523,50cyclictest12-21ksoftirqd/009:52:520
2859117998421,13cyclictest2898053-21ssh10:19:220
2859117998329,37cyclictest2873778-21ssh09:27:440
2859117998321,15cyclictest2920271-21ssh11:06:080
285912899814,66cyclictest48-21kcompactd008:50:033
285911799812,24cyclictest2953935-21ssh12:17:500
285911799801,62cyclictest2941580-21ssh11:51:360
285911799801,62cyclictest2941580-21ssh11:51:360
2859117997924,51cyclictest2938748-21ssh11:45:260
285911799791,60cyclictest0-21swapper/011:00:520
2859117997874,2cyclictest0-21swapper/009:40:030
2859117997829,32cyclictest2889739-21ssh10:01:270
2859117997735,37cyclictest2872397-21systemd-run09:25:000
2859117997727,33cyclictest2887532-21ssh09:56:510
2859117997722,50cyclictest2945686-21ssh12:00:030
2859117997722,37cyclictest2915310-21ssh10:55:430
2859117997715,6cyclictest16850irq/134-800000010:47:550
2859117997628,44cyclictest2951694-21ssh12:13:100
285911799762,69cyclictest2933528-21ssh11:34:490
2859117997622,39cyclictest2892203-21ssh10:06:390
285911799761,70cyclictest0-21swapper/012:25:020
285911799761,57cyclictest0-21swapper/009:13:470
2859117997525,46cyclictest2878685-21ssh09:38:130
2859117997429,6cyclictest2861904-21cron08:05:000
285911799733,11cyclictest2877116-21latency_hist09:35:010
2859117997325,31cyclictest2922751-21ssh11:11:320
285911799725,60cyclictest2926542-21modprobe11:20:000
285911799721,57cyclictest0-21swapper/012:38:150
285911799712,19cyclictest3467602-21systemd-journal07:15:010
2859117997115,16cyclictest2883600-21ssh09:48:340
285911799711,4cyclictest2934896-21ssh11:37:300
285912299707,7cyclictest23-21ksoftirqd/111:05:011
2859117997027,39cyclictest2929190-21ssh11:25:080
2859117997027,26cyclictest2869998-21ssh09:19:540
2859117997020,5cyclictest0-21swapper/012:28:010
2859117997020,5cyclictest0-21swapper/012:28:000
285911799701,53cyclictest0-21swapper/010:51:420
2859117996819,4cyclictest0-21swapper/010:32:170
2859117996722,28cyclictest2906036-21ssh10:36:080
2859126996616,10cyclictest30-21ksoftirqd/211:20:022
285911799668,14cyclictest2961849-21ssh12:34:420
2859117996539,23cyclictest12-21ksoftirqd/010:41:460
2859117996537,25cyclictest16950irq/224-800000010:30:000
2859117996526,33cyclictest16950irq/224-800000007:55:010
285911799651,16cyclictest2860846-21sshd07:40:170
2859126996455,6cyclictest30-21ksoftirqd/210:35:022
2859117996416,6cyclictest16850irq/134-800000010:23:380
2859117996416,6cyclictest16850irq/134-800000010:23:380
285912699638,21cyclictest2909745-21ssh10:44:092
2859126996318,10cyclictest30-21ksoftirqd/211:34:562
2859126996315,13cyclictest2884130-21ssh09:49:472
2859126996310,16cyclictest2913994-21ssh10:53:072
2859117996320,38cyclictest309-21in:imuxsock10:15:000
285911799631,58cyclictest0-21swapper/012:05:500
2859126996218,8cyclictest30-21ksoftirqd/212:17:562
2859126996216,10cyclictest30-21ksoftirqd/212:07:492
2859122996211,12cyclictest23-21ksoftirqd/109:15:001
2859117996227,31cyclictest2864732-21sshd08:59:170
2859117996216,12cyclictest2927376-21ssh11:21:190
2859126996115,10cyclictest30-21ksoftirqd/210:14:252
285911799611,56cyclictest0-21swapper/008:44:170
2859117996113,6cyclictest2863704-21cron08:40:000
2859126996015,9cyclictest30-21ksoftirqd/209:12:112
2859126996014,8cyclictest30-21ksoftirqd/211:40:032
2859126996013,11cyclictest30-21ksoftirqd/209:15:202
285911799603,12cyclictest2862413-21modprobe08:15:000
285911799602,15cyclictest2862773-21sort08:20:020
285911799602,15cyclictest2862773-21sort08:20:020
2859117996019,13cyclictest2944146-21ssh11:57:040
2859126995916,8cyclictest30-21ksoftirqd/209:28:242
2859126995914,10cyclictest30-21ksoftirqd/211:21:282
2859122995928,8cyclictest23-21ksoftirqd/111:36:381
2859122995912,10cyclictest23-21ksoftirqd/112:15:021
2859128995837,16cyclictest3467602-21systemd-journal11:35:003
285912699586,20cyclictest2871192-21sh09:22:062
2859126995814,11cyclictest30-21ksoftirqd/212:36:372
285912899571,10cyclictest0-21swapper/310:43:183
285912699576,16cyclictest2930314-21ssh11:27:392
2859126995715,10cyclictest30-21ksoftirqd/210:48:122
2859126995714,10cyclictest30-21ksoftirqd/212:05:022
285912299577,5cyclictest131rcu_preempt09:40:081
2859117995735,16cyclictest3467602-21systemd-journal08:25:000
285912699569,15cyclictest2944005-21ssh11:56:452
285912699566,18cyclictest2937709-21ssh11:43:252
285912699566,18cyclictest2937709-21ssh11:43:252
285912699566,14cyclictest2907654-21munin-run10:40:002
2859126995610,8cyclictest2919288-21ssh11:04:212
285912299565,9cyclictest23-21ksoftirqd/111:45:021
285912299565,9cyclictest23-21ksoftirqd/111:45:021
285911799562,17cyclictest2863462-21munin-plugin-st08:35:000
2859117995616,4cyclictest2863352-21latency_hist08:30:020
2859117995612,9cyclictest275-21dbus-daemon08:00:000
285912699558,14cyclictest2879317-21ssh09:39:442
2859126995511,10cyclictest30-21ksoftirqd/212:33:292
2859117995522,29cyclictest2860610-21latency_hist07:40:020
285911799552,17cyclictest2859846-21munin-run07:25:000
2859117995518,10cyclictest2860170-21date07:30:020
285912699544,17cyclictest2900218-21ssh10:23:562
285912699544,17cyclictest2900218-21ssh10:23:552
2859126995414,10cyclictest30-21ksoftirqd/212:11:122
2859126995413,9cyclictest30-21ksoftirqd/211:14:442
2859126995410,7cyclictest2863755-21systemctl08:40:002
285912299547,7cyclictest23-21ksoftirqd/109:22:141
285912299545,13cyclictest23-21ksoftirqd/109:54:431
285912299545,10cyclictest23-21ksoftirqd/112:25:021
2859122995440,9cyclictest23-21ksoftirqd/110:43:191
2859122995418,7cyclictest131rcu_preempt11:25:001
2859122995415,10cyclictest23-21ksoftirqd/110:40:011
285911799543,12cyclictest2859578-21cut07:20:000
285912699534,17cyclictest2860144-21cat07:30:022
2859126995312,9cyclictest30-21ksoftirqd/212:27:392
2859126995312,9cyclictest30-21ksoftirqd/212:27:392
285912299538,9cyclictest23-21ksoftirqd/110:00:001
285912299538,12cyclictest23-21ksoftirqd/112:09:281
285912299537,12cyclictest23-21ksoftirqd/110:06:001
285912299536,12cyclictest23-21ksoftirqd/111:26:271
2859122995312,5cyclictest131rcu_preempt09:30:011
2859122995310,6cyclictest131rcu_preempt11:05:041
285911799538,4cyclictest131rcu_preempt07:35:000
285912699526,16cyclictest2943047-21munin-run11:55:002
285912699526,16cyclictest2943047-21munin-run11:55:002
2859126995212,7cyclictest30-21ksoftirqd/211:06:252
2859126995212,14cyclictest2938752-21ssh11:45:272
2859126995211,11cyclictest30-21ksoftirqd/209:56:422
285912299528,6cyclictest131rcu_preempt11:33:371
285912299528,6cyclictest131rcu_preempt09:30:311
285912299528,5cyclictest131rcu_preempt10:59:521
285912299527,8cyclictest23-21ksoftirqd/110:52:011
285912299527,7cyclictest23-21ksoftirqd/111:47:241
2859122995235,12cyclictest275-21dbus-daemon08:25:001
2859122995210,5cyclictest131rcu_preempt12:32:081
2859117995221,28cyclictest2861916-21latency_hist08:05:020
2859117995221,28cyclictest2861916-21latency_hist08:05:020
285912699518,9cyclictest210-21jbd2/mmcblk1p2-07:35:012
2859126995111,8cyclictest30-21ksoftirqd/209:45:012
285912299519,5cyclictest131rcu_preempt10:25:281
285912299512,10cyclictest23-21ksoftirqd/110:17:051
2859122995118,6cyclictest131rcu_preempt09:00:011
2859122995111,5cyclictest131rcu_preempt09:36:031
2859117995128,18cyclictest275-21dbus-daemon08:50:010
2859126995011,14cyclictest2889250-21sh10:00:152
285912299507,5cyclictest131rcu_preempt12:01:231
285912299503,8cyclictest23-21ksoftirqd/110:21:171
285912299503,8cyclictest23-21ksoftirqd/110:21:171
2859122995032,13cyclictest3467602-21systemd-journal12:29:181
2859122995032,13cyclictest3467602-21systemd-journal12:29:171
2859122995010,5cyclictest131rcu_preempt10:02:081
2859117995021,8cyclictest12-21ksoftirqd/007:10:020
285912899499,5cyclictest131rcu_preempt11:25:033
2859126994941,4cyclictest2861676-21kworker/2:110:56:542
2859126994934,12cyclictest2876725-21ssh09:34:072
285912299496,10cyclictest23-21ksoftirqd/110:13:401
285912299495,7cyclictest23-21ksoftirqd/112:16:531
285912299495,13cyclictest23-21ksoftirqd/109:20:001
285912299494,12cyclictest23-21ksoftirqd/110:33:291
2859122994935,10cyclictest23-21ksoftirqd/111:55:211
285912299493,11cyclictest23-21ksoftirqd/111:13:421
2859122994912,6cyclictest131rcu_preempt11:51:001
2859122994912,6cyclictest131rcu_preempt11:51:001
2859122994912,11cyclictest23-21ksoftirqd/107:50:011
2859128994836,9cyclictest2925851-21sshd11:18:173
2859128994833,11cyclictest3467602-21systemd-journal12:35:003
2859128994814,4cyclictest2868747-21cron09:17:003
2859128994810,34cyclictest1-21systemd10:00:003
2859126994813,7cyclictest0-21swapper/209:54:452
285912299489,5cyclictest131rcu_preempt09:49:211
285912299485,8cyclictest23-21ksoftirqd/110:45:391
285912299485,10cyclictest23-21ksoftirqd/108:30:001
285912299484,8cyclictest23-21ksoftirqd/112:39:261
2859128994734,8cyclictest2907652-21munin-plugin-st10:40:003
285912699478,8cyclictest30-21ksoftirqd/210:16:172
285912299474,38cyclictest3467602-21systemd-journal08:01:171
2859122994739,6cyclictest23-21ksoftirqd/108:50:021
2859122994737,7cyclictest221rcuc/111:15:511
285912899463,3cyclictest131rcu_preempt12:20:183
2859128994629,13cyclictest2881867-21cron09:45:003
2859128994627,8cyclictest2860576-21sshd07:37:173
2859126994635,5cyclictest309-21in:imuxsock07:15:012
2859126994634,9cyclictest2956407-21ssh12:23:082
2859126994631,12cyclictest2864762-21systemctl09:00:002
2859126994610,7cyclictest30-21ksoftirqd/210:09:572
285912299469,11cyclictest23-21ksoftirqd/107:20:001
285912299467,8cyclictest23-21ksoftirqd/107:30:011
285912299462,7cyclictest23-21ksoftirqd/108:40:011
285912299461,8cyclictest23-21ksoftirqd/109:10:011
285912299461,16cyclictest0-21swapper/109:02:261
285911799463,14cyclictest2864499-21munin-run08:55:000
2859128994531,10cyclictest275-21dbus-daemon09:55:023
2859128994529,13cyclictest37-21ksoftirqd/310:30:023
285912899452,7cyclictest0-21swapper/311:36:333
2859126994537,5cyclictest2860909-21kworker/2:308:00:512
2859126994529,12cyclictest275-21dbus-daemon08:35:022
285912299457,11cyclictest23-21ksoftirqd/108:54:171
285912299454,7cyclictest23-21ksoftirqd/107:25:021
285912899449,6cyclictest131rcu_preempt07:50:013
2859128994429,12cyclictest3467602-21systemd-journal08:32:023
2859128994429,10cyclictest1-21systemd11:05:013
285912899441,8cyclictest3467602-21systemd-journal11:11:173
285912899441,7cyclictest0-21swapper/310:53:513
285912899441,3cyclictest131rcu_preempt11:57:463
2859126994432,10cyclictest2901333-21ssh10:26:062
285912299446,11cyclictest23-21ksoftirqd/108:09:171
285912299446,11cyclictest23-21ksoftirqd/108:09:171
285912899437,6cyclictest131rcu_preempt10:00:543
2859128994328,11cyclictest3467602-21systemd-journal10:35:013
2859128994326,11cyclictest2575505-1kworker/u9:109:40:033
2859126994334,6cyclictest2860607-21sh07:40:002
2859126994334,5cyclictest2861676-21kworker/2:108:18:052
2859126994334,5cyclictest2861676-21kworker/2:108:18:052
285912299436,8cyclictest23-21ksoftirqd/108:45:021
285912299431,6cyclictest0-21swapper/107:14:241
285912299431,5cyclictest0-21swapper/107:40:011
2859128994233,5cyclictest2964298-21munin-plugin-st12:40:003
2859128994232,7cyclictest37-21ksoftirqd/312:14:483
2859128994232,7cyclictest2892022-21sshd10:06:173
2859128994229,9cyclictest37-21ksoftirqd/309:00:023
2859128994228,10cyclictest275-21dbus-daemon10:15:013
2859128994224,7cyclictest37-21ksoftirqd/310:20:003
285912899422,28cyclictest2853256-21kworker/u8:207:25:023
2859126994227,12cyclictest2863379-21sed08:30:022
2859126994210,24cyclictest2862382-21modprobe08:10:032
2859122994214,9cyclictest23-21ksoftirqd/108:20:011
2859122994214,9cyclictest23-21ksoftirqd/108:20:011
2859128994133,6cyclictest37-21ksoftirqd/310:45:513
2859128994128,9cyclictest37-21ksoftirqd/309:29:273
2859128994128,9cyclictest2864994-21sshd09:00:173
2859128994127,10cyclictest37-21ksoftirqd/309:21:433
285912899412,6cyclictest131rcu_preempt08:40:003
2859128994125,12cyclictest37-21ksoftirqd/312:27:193
2859128994125,12cyclictest37-21ksoftirqd/312:27:183
2859128994124,13cyclictest1-21systemd09:10:013
285912899411,5cyclictest131rcu_preempt10:55:083
285912699414,7cyclictest30-21ksoftirqd/207:19:252
2859126994129,7cyclictest2863239-21kworker/2:009:05:022
2859126994118,18cyclictest2865296-21ssh09:10:012
2859126994113,5cyclictest131rcu_preempt08:25:012
285912299418,8cyclictest23-21ksoftirqd/107:57:341
285912299417,8cyclictest23-21ksoftirqd/107:45:011
285912299413,12cyclictest23-21ksoftirqd/108:15:001
285912899409,6cyclictest131rcu_preempt12:01:533
285912899403,7cyclictest131rcu_preempt08:20:003
285912899403,7cyclictest131rcu_preempt08:20:003
285912899403,5cyclictest131rcu_preempt08:45:023
285912899403,34cyclictest2859525-21kworker/u8:007:20:023
2859128994030,6cyclictest2938408-21grep11:45:003
2859128994030,6cyclictest2938408-21grep11:45:003
285912899402,8cyclictest131rcu_preempt10:24:433
285912899402,8cyclictest131rcu_preempt10:24:433
2859128994026,11cyclictest37-21ksoftirqd/311:08:383
2859126994026,10cyclictest48-21kcompactd007:55:072
2859126994023,13cyclictest2864318-21cat08:50:022
2859122994025,10cyclictest3467602-21systemd-journal08:30:021
2859117994018,4cyclictest0-21swapper/009:02:260
285911799401,24cyclictest0-21swapper/007:45:360
285912899398,5cyclictest131rcu_preempt12:10:013
2859128993930,5cyclictest2863209-21munin-plugin-st08:30:003
2859128993929,5cyclictest309-21in:imuxsock08:00:003
2859128993928,8cyclictest37-21ksoftirqd/307:45:023
2859128993927,9cyclictest37-21ksoftirqd/312:18:173
2859128993916,19cyclictest37-21ksoftirqd/309:10:503
2859128993915,6cyclictest131rcu_preempt11:50:223
2859128993915,6cyclictest131rcu_preempt11:50:213
2859126993926,10cyclictest2864466-21sshd08:51:172
2859126993924,12cyclictest2861098-21sshd07:45:172
2859128993826,9cyclictest2931339-21sed11:30:023
285912899381,5cyclictest131rcu_preempt09:34:353
2859128993815,5cyclictest37-21ksoftirqd/311:47:573
285912699388,9cyclictest0-21swapper/207:52:382
2859126993825,9cyclictest48-21kcompactd007:21:172
2859126993825,10cyclictest2860856-21sshd07:42:172
2859126993822,11cyclictest2862182-21munin-run08:10:012
2859126993822,11cyclictest2862182-21munin-run08:10:012
285912299388,6cyclictest131rcu_preempt07:50:171
285912299381,8cyclictest23-21ksoftirqd/107:30:431
285912299381,7cyclictest2859243-21cut07:10:021
2859128993724,10cyclictest37-21ksoftirqd/307:15:023
2859128993722,12cyclictest37-21ksoftirqd/307:20:003
2859128993714,4cyclictest37-21ksoftirqd/309:49:503
2859126993713,19cyclictest2864097-21latency_hist08:45:022
2859128993621,11cyclictest3467602-21systemd-journal08:25:003
2859128993524,9cyclictest37-21ksoftirqd/308:05:023
2859128993524,9cyclictest37-21ksoftirqd/308:05:023
2859128993522,10cyclictest37-21ksoftirqd/307:35:013
285912899343,9cyclictest37-21ksoftirqd/308:14:563
2859128993425,7cyclictest37-21ksoftirqd/307:10:023
285912899341,4cyclictest131rcu_preempt08:50:473
2859128993225,4cyclictest3467562-21sshd08:01:173
2859128993221,7cyclictest0-21swapper/307:55:013
2859126992820,5cyclictest2859196-21wc07:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional