You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-30 - 10:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Mar 30, 2026 00:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3140539999122,15cyclictest3173118-21ssh22:11:450
3140539998831,52cyclictest3224610-21ssh00:13:030
3140539998624,14cyclictest3162044-21ssh21:45:260
314053999841,65cyclictest3166244-21ssh21:55:220
3140539998128,49cyclictest3160202-21ssh21:41:120
3140539998128,49cyclictest3160202-21ssh21:41:110
314053999811,2cyclictest0-21swapper/022:25:010
314053999811,2cyclictest0-21swapper/022:25:000
3140539997927,48cyclictest3203908-21ssh23:24:360
3140539997926,48cyclictest3219484-21ssh00:00:360
3140539997926,48cyclictest3219484-21ssh00:00:350
3140539997922,16cyclictest3176473-21ssh22:19:540
314053999791,73cyclictest3184502-21ssh22:38:410
314053999782,60cyclictest3236014-21ssh00:39:470
3140539997728,32cyclictest3156892-21ssh21:33:440
3140539997724,12cyclictest3209900-21ssh23:38:290
3140539997629,30cyclictest3187959-21ssh22:46:440
3140539997629,30cyclictest3187959-21ssh22:46:430
3140539997627,31cyclictest3228184-21ssh00:21:110
3140539997627,31cyclictest3228184-21ssh00:21:110
314053999762,50cyclictest3172281-21sh22:10:020
3140539997528,29cyclictest3211264-21ssh23:41:290
3140539997526,32cyclictest3159366-21ssh21:39:390
3140539997525,14cyclictest3152833-21ssh21:24:160
3140539997426,30cyclictest3214598-21ssh23:49:360
3140539997325,15cyclictest3196588-21ssh23:07:050
3140539997325,15cyclictest3196588-21ssh23:07:040
314053999731,26cyclictest0-21swapper/021:53:530
3140539997224,2cyclictest0-21swapper/021:28:230
3140539997213,47cyclictest12-21ksoftirqd/021:15:010
314053999719,57cyclictest3202170-21ssh23:20:030
314053999719,57cyclictest3202170-21ssh23:20:020
3140539997126,28cyclictest3206113-21ssh23:29:530
3140539997124,42cyclictest309-21in:imuxsock21:05:000
3140545997062,5cyclictest3144925-21kworker/2:021:40:012
314053999702,18cyclictest3216281-21ssh23:53:300
314053999701,4cyclictest0-21swapper/022:34:060
3140539996918,14cyclictest3179771-21ssh22:27:260
314053999691,63cyclictest3168814-21ssh22:01:270
314053999691,63cyclictest3168814-21ssh22:01:270
314054599687,7cyclictest131rcu_preempt23:05:022
3140539996820,13cyclictest3198355-21ssh23:11:110
314053999672,5cyclictest3208329-21munin-run23:35:000
3140539996723,13cyclictest3185985-21ssh22:41:590
3140539996721,14cyclictest3191671-21ssh22:55:130
3140543996660,3cyclictest231ksoftirqd/100:10:011
3140539996634,29cyclictest3194418-21sshd23:01:540
3140539996629,31cyclictest3218976-40mandb00:00:000
3140539996623,11cyclictest3149751-21ssh21:16:470
3140539996619,8cyclictest1-21systemd22:51:510
3140539996522,38cyclictest3143841-21cron20:15:010
3140539996522,15cyclictest3230859-21ssh00:27:300
314053999641,49cyclictest0-21swapper/019:40:010
3140539996321,10cyclictest309-21in:imuxsock00:35:000
3140539996219,6cyclictest16850irq/134-800000000:16:320
3140543996137,15cyclictest1-21systemd00:00:001
3140539996020,7cyclictest0-21swapper/000:09:120
314053999601,26cyclictest48-21kcompactd020:57:090
3140543995950,6cyclictest23-21ksoftirqd/121:15:021
3140543995631,19cyclictest275-21dbus-daemon00:30:001
3140539995516,9cyclictest12-21ksoftirqd/020:10:010
314054399546,38cyclictest3467602-21systemd-journal23:35:001
3140539995420,10cyclictest12-21ksoftirqd/019:15:030
3140545995345,5cyclictest3159561-21kworker/2:222:40:052
3140545995344,5cyclictest3159561-21kworker/2:222:00:022
3140543995335,9cyclictest309-21in:imuxsock19:30:001
314054399531,4cyclictest131rcu_preempt22:05:011
314054399531,4cyclictest131rcu_preempt22:05:011
3140545995245,4cyclictest3163777-21kworker/2:100:06:512
3140545995244,5cyclictest3163777-21kworker/2:123:47:202
3140543995239,8cyclictest3467602-21systemd-journal19:55:001
3140543995231,15cyclictest3467602-21systemd-journal00:35:011
3140543995227,19cyclictest3208998-21md.daily23:36:151
3140543995210,26cyclictest3142029-21modprobe19:40:011
3140539995244,6cyclictest12-21ksoftirqd/019:34:380
3140550995115,6cyclictest131rcu_preempt22:34:003
3140545995143,5cyclictest3159561-21kworker/2:200:24:112
3140545995143,5cyclictest3159561-21kworker/2:200:24:102
3140545995143,5cyclictest3144925-21kworker/2:021:22:032
3140545995140,7cyclictest3140500-21kworker/2:120:59:592
314054399517,6cyclictest131rcu_preempt22:50:011
314054399517,6cyclictest131rcu_preempt22:50:011
3140539995135,12cyclictest1-21systemd20:35:010
3140539995123,4cyclictest0-21swapper/019:22:430
3140539995123,4cyclictest0-21swapper/019:22:420
314055099502,11cyclictest3162580-21ssh21:46:513
3140545995042,5cyclictest3163777-21kworker/2:100:14:012
3140545995042,5cyclictest3151063-21kworker/2:321:34:432
3140545995041,5cyclictest3163777-21kworker/2:122:32:592
3140545995040,7cyclictest3144925-21kworker/2:021:15:502
3140545995038,9cyclictest3467602-21systemd-journal20:05:022
3140543995033,13cyclictest1-21systemd21:10:011
3140543995019,21cyclictest23-21ksoftirqd/122:58:081
3140539995015,7cyclictest3143114-21unin-run20:00:020
3140550994933,11cyclictest275-21dbus-daemon20:35:013
3140550994932,13cyclictest275-21dbus-daemon23:00:003
3140545994941,5cyclictest3163777-21kworker/2:122:56:302
3140545994941,5cyclictest3159561-21kworker/2:223:41:282
3140545994941,5cyclictest3159561-21kworker/2:223:30:202
3140545994941,5cyclictest3159561-21kworker/2:222:46:272
3140545994941,5cyclictest3159561-21kworker/2:222:46:272
3140545994941,5cyclictest3159561-21kworker/2:221:51:562
3140545994940,5cyclictest3151063-21kworker/2:321:29:062
314054599493,5cyclictest131rcu_preempt21:10:022
3140543994929,16cyclictest275-21dbus-daemon21:35:011
314054399492,5cyclictest131rcu_preempt22:40:021
314053999497,30cyclictest3140542-21latency_hist19:10:020
3140539994913,10cyclictest364-21snmpd19:41:150
3140550994835,8cyclictest1-21systemd22:00:013
3140545994840,5cyclictest3227564-21kworker/2:300:30:152
3140545994839,6cyclictest3159561-21kworker/2:221:40:572
3140545994839,6cyclictest3159561-21kworker/2:221:40:572
3140545994831,13cyclictest1-21systemd19:55:012
314054599481,43cyclictest3467602-21systemd-journal19:37:372
314054399483,6cyclictest131rcu_preempt22:41:471
3140543994822,18cyclictest275-21dbus-daemon22:35:011
3140543994821,7cyclictest131rcu_preempt21:30:011
314055099473,4cyclictest131rcu_preempt22:45:493
314055099473,4cyclictest131rcu_preempt22:45:483
3140545994739,5cyclictest3163777-21kworker/2:123:23:312
3140545994739,5cyclictest3163777-21kworker/2:122:25:002
3140545994739,5cyclictest3163777-21kworker/2:122:24:592
3140545994739,5cyclictest3163777-21kworker/2:122:17:232
3140545994739,5cyclictest3163777-21kworker/2:100:02:172
3140545994739,5cyclictest3163777-21kworker/2:100:02:172
3140545994739,5cyclictest3159561-21kworker/2:223:08:172
3140545994739,5cyclictest3159561-21kworker/2:223:08:172
3140545994739,5cyclictest3133475-21kworker/2:219:17:452
3140545994733,10cyclictest3467602-21systemd-journal20:27:362
314054399475,5cyclictest131rcu_preempt23:30:001
3140539994716,7cyclictest3144565-21date20:25:030
3140539994716,7cyclictest3144565-21date20:25:020
3140539994713,13cyclictest3146786-21rm21:10:020
314055099463,3cyclictest131rcu_preempt00:28:593
314055099462,3cyclictest131rcu_preempt21:16:463
3140545994638,5cyclictest3227564-21kworker/2:300:28:382
3140545994638,5cyclictest3163777-21kworker/2:123:57:002
3140545994638,5cyclictest3163777-21kworker/2:123:11:572
3140545994638,5cyclictest3159561-21kworker/2:222:38:362
3140545994631,12cyclictest30-21ksoftirqd/223:27:152
3140543994635,8cyclictest23-21ksoftirqd/121:50:001
3140543994635,8cyclictest23-21ksoftirqd/100:24:231
3140543994635,8cyclictest23-21ksoftirqd/100:24:231
3140543994633,9cyclictest23-21ksoftirqd/121:55:181
3140543994628,13cyclictest3467602-21systemd-journal00:19:591
3140543994627,14cyclictest275-21dbus-daemon23:45:011
3140543994621,20cyclictest23-21ksoftirqd/123:06:561
3140543994621,20cyclictest23-21ksoftirqd/123:06:561
3140539994615,8cyclictest12-21ksoftirqd/019:30:020
3140550994531,11cyclictest3214731-21sh23:50:013
3140550994526,4cyclictest37-21ksoftirqd/300:00:003
314055099451,4cyclictest131rcu_preempt21:31:573
3140550994511,6cyclictest0-21swapper/300:09:043
314054599454,4cyclictest131rcu_preempt21:12:212
3140545994537,5cyclictest3163777-21kworker/2:122:14:272
3140545994537,5cyclictest3159561-21kworker/2:222:52:562
3140543994530,11cyclictest23-21ksoftirqd/123:21:311
3140543994519,16cyclictest275-21dbus-daemon19:25:011
3140543994519,16cyclictest275-21dbus-daemon19:25:011
3140539994525,16cyclictest3467602-21systemd-journal20:45:010
314055099444,3cyclictest131rcu_preempt23:01:483
3140550994428,12cyclictest3467602-21systemd-journal21:30:013
314055099442,3cyclictest131rcu_preempt00:12:583
314055099441,6cyclictest3148869-21latency_hist21:15:023
3140545994436,5cyclictest3163777-21kworker/2:122:09:022
3140545994436,5cyclictest3159561-21kworker/2:223:35:402
3140545994436,5cyclictest3159561-21kworker/2:222:02:282
3140545994436,5cyclictest3159561-21kworker/2:222:02:272
3140543994433,8cyclictest3143043-21sshd19:57:381
3140543994433,7cyclictest3221258-21cat00:05:021
3140543994433,7cyclictest3221258-21cat00:05:011
3140543994422,8cyclictest23-21ksoftirqd/122:29:591
3140539994437,4cyclictest16950irq/224-800000019:49:590
3140539994432,8cyclictest3145158-21latency_hist20:40:010
314053999441,24cyclictest0-21swapper/020:53:130
314053999441,24cyclictest0-21swapper/020:53:120
3140550994333,8cyclictest3234010-21systemd-run00:35:013
3140550994330,10cyclictest3467602-21systemd-journal21:43:363
3140550994330,10cyclictest3467602-21systemd-journal21:43:363
314055099432,6cyclictest131rcu_preempt23:39:593
314054599434,3cyclictest131rcu_preempt20:10:022
3140545994335,5cyclictest3141286-21kworker/2:320:34:332
314054399439,6cyclictest131rcu_preempt20:15:011
314054399438,24cyclictest3176526-21modprobe22:20:001
314054399433,5cyclictest131rcu_preempt20:55:011
314054399433,5cyclictest131rcu_preempt20:55:001
3140543994328,12cyclictest23-21ksoftirqd/119:50:021
3140543994328,11cyclictest23-21ksoftirqd/121:00:001
3140543994328,10cyclictest3158584-21sshd21:37:361
3140543994327,12cyclictest23-21ksoftirqd/119:19:591
3140543994319,19cyclictest23-21ksoftirqd/122:10:321
3140543994316,22cyclictest221rcuc/121:20:291
314055099429,5cyclictest131rcu_preempt22:40:143
314055099429,5cyclictest131rcu_preempt21:51:223
314055099428,5cyclictest131rcu_preempt23:14:003
314055099428,4cyclictest131rcu_preempt22:28:593
314055099426,5cyclictest131rcu_preempt22:23:373
314055099426,5cyclictest131rcu_preempt22:23:373
3140550994225,12cyclictest3201975-21munin-run23:20:003
3140550994225,12cyclictest3201975-21munin-run23:20:003
314055099422,3cyclictest131rcu_preempt22:36:483
314055099422,3cyclictest131rcu_preempt22:04:413
314055099422,3cyclictest131rcu_preempt22:04:403
3140550994211,5cyclictest131rcu_preempt22:52:113
3140545994236,3cyclictest3159561-21kworker/2:221:48:022
3140545994234,5cyclictest3227564-21kworker/2:300:36:102
3140545994234,5cyclictest3140500-21kworker/2:120:16:372
3140545994231,8cyclictest3467602-21systemd-journal20:22:382
3140545994231,8cyclictest3467602-21systemd-journal20:22:372
3140545994230,9cyclictest30-21ksoftirqd/223:53:232
314054599422,9cyclictest310-21in:imklog19:15:012
314054599422,9cyclictest310-21in:imklog19:15:012
314054399427,6cyclictest131rcu_preempt00:40:011
314054399427,25cyclictest3142307-21modprobe19:44:591
3140543994233,5cyclictest3146720-21diskmemload23:20:021
3140543994233,5cyclictest3146720-21diskmemload23:20:021
3140543994230,9cyclictest23-21ksoftirqd/100:10:171
3140543994230,10cyclictest23-21ksoftirqd/122:50:111
3140543994229,9cyclictest309-21in:imuxsock20:40:011
3140543994226,13cyclictest23-21ksoftirqd/121:19:361
3140543994218,4cyclictest23-21ksoftirqd/122:08:191
314054399421,13cyclictest346771850lldpd20:26:201
314053999421,25cyclictest0-21swapper/020:02:390
314055099419,5cyclictest131rcu_preempt22:11:463
314055099413,3cyclictest131rcu_preempt00:17:113
3140550994129,7cyclictest3144407-21unin-run20:25:013
3140550994129,7cyclictest3144407-21unin-run20:25:013
3140550994127,10cyclictest275-21dbus-daemon20:20:003
3140550994125,10cyclictest3467602-21systemd-journal20:05:003
3140550994121,4cyclictest37-21ksoftirqd/323:31:523
314054599413,4cyclictest131rcu_preempt00:16:182
3140545994133,5cyclictest3163777-21kworker/2:122:29:522
3140545994131,6cyclictest3129873-21kworker/2:019:10:022
3140545994130,8cyclictest30-21ksoftirqd/219:30:022
314054599411,4cyclictest131rcu_preempt20:00:002
314054399416,11cyclictest23-21ksoftirqd/120:25:011
314054399416,11cyclictest23-21ksoftirqd/120:25:001
314055099403,3cyclictest131rcu_preempt22:19:013
3140550994026,9cyclictest275-21dbus-daemon23:55:003
3140550994026,10cyclictest3143840-21cron20:15:013
3140550994026,10cyclictest275-21dbus-daemon21:25:013
314055099402,3cyclictest131rcu_preempt20:00:033
3140550994023,11cyclictest275-21dbus-daemon20:10:013
3140550994022,13cyclictest1-21systemd19:45:013
3140550994021,5cyclictest37-21ksoftirqd/323:27:333
3140550994011,3cyclictest131rcu_preempt00:24:053
3140550994011,3cyclictest131rcu_preempt00:24:053
3140545994032,5cyclictest3140500-21kworker/2:119:43:562
3140545994026,10cyclictest275-21dbus-daemon19:35:002
314054599402,3cyclictest131rcu_preempt19:25:012
314054599402,3cyclictest131rcu_preempt19:25:002
314054599401,4cyclictest131rcu_preempt20:55:022
314054599401,4cyclictest131rcu_preempt20:55:012
314054399406,6cyclictest131rcu_preempt20:10:001
314054399403,31cyclictest3195587-21munin-plugin-st23:05:001
314054399402,6cyclictest131rcu_preempt23:54:271
314054399401,5cyclictest131rcu_preempt19:10:011
3140539994033,4cyclictest16950irq/224-800000019:55:010
3140539994025,11cyclictest12-21ksoftirqd/020:16:590
314055099393,4cyclictest131rcu_preempt23:45:023
3140550993925,10cyclictest37-21ksoftirqd/323:22:323
3140550993924,11cyclictest3467602-21systemd-journal20:30:013
314054399395,5cyclictest131rcu_preempt22:25:031
314054399395,5cyclictest131rcu_preempt22:25:031
3140543993925,10cyclictest23-21ksoftirqd/123:14:551
3140543993925,10cyclictest23-21ksoftirqd/121:45:001
3140543993925,10cyclictest23-21ksoftirqd/121:44:591
3140539993914,17cyclictest0-21swapper/019:15:010
3140539993914,17cyclictest0-21swapper/019:15:000
3140550993829,5cyclictest309-21in:imuxsock19:20:003
3140550993825,11cyclictest37-21ksoftirqd/321:39:333
314055099382,4cyclictest131rcu_preempt00:03:303
314055099382,4cyclictest131rcu_preempt00:03:293
3140550993823,11cyclictest37-21ksoftirqd/323:05:233
3140550993823,11cyclictest37-21ksoftirqd/323:05:233
3140550993823,11cyclictest3467602-21systemd-journal21:10:013
3140550993821,12cyclictest275-21dbus-daemon19:40:003
314055099381,4cyclictest131rcu_preempt19:50:023
314055099381,3cyclictest131rcu_preempt00:36:073
3140545993828,8cyclictest30-21ksoftirqd/223:15:262
3140545993828,8cyclictest30-21ksoftirqd/223:15:262
3140545993825,10cyclictest3146525-21wc21:05:022
3140543993832,3cyclictest231ksoftirqd/121:55:011
314054399382,6cyclictest131rcu_preempt20:45:001
3140543993824,11cyclictest23-21ksoftirqd/123:50:011
3140539993831,4cyclictest16950irq/224-800000020:50:020
3140550993726,9cyclictest37-21ksoftirqd/320:45:003
3140550993726,8cyclictest309-21in:imuxsock19:11:383
3140550993726,8cyclictest309-21in:imuxsock19:11:383
3140545993725,8cyclictest3145153-21cron20:40:002
3140545993724,9cyclictest30-21ksoftirqd/220:12:592
314054599371,3cyclictest131rcu_preempt20:50:012
314054399379,4cyclictest131rcu_preempt20:05:001
314054399374,7cyclictest131rcu_preempt21:05:011
314053999372,31cyclictest3144705-21latency_hist20:30:000
314055099363,4cyclictest131rcu_preempt22:08:283
3140550993629,4cyclictest3145167-21sendmail20:40:013
3140550993623,10cyclictest37-21ksoftirqd/319:30:013
3140550993617,14cyclictest275-21dbus-daemon21:00:013
314054399365,20cyclictest3140727-21modprobe19:15:011
314054399365,20cyclictest3140727-21modprobe19:15:011
3140543993622,10cyclictest23-21ksoftirqd/120:50:011
314054399361,8cyclictest131rcu_preempt20:20:021
3140545993524,8cyclictest30-21ksoftirqd/219:45:172
314054599352,3cyclictest131rcu_preempt20:45:012
3140543993422,10cyclictest23-21ksoftirqd/120:30:361
3140550993322,8cyclictest1-21systemd19:25:013
3140550993322,8cyclictest1-21systemd19:25:013
314054399333,22cyclictest3141779-21munin-run19:35:001
314055099322,4cyclictest131rcu_preempt20:50:003
314055099314,4cyclictest131rcu_preempt21:04:373
314055099312,4cyclictest131rcu_preempt20:55:013
314055099312,4cyclictest131rcu_preempt20:55:003
3140550993118,10cyclictest3140563-21sed19:10:013
3140550993118,10cyclictest1-21systemd19:35:023
314055099301,3cyclictest131rcu_preempt19:55:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional