You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-26 - 23:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun Jul 26, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3199190991131,109cyclictest0-21swapper/007:25:030
319919099112108,2cyclictest0-21swapper/009:55:010
3199195998948,24cyclictest131rcu_preempt09:40:031
319919099883,10cyclictest131rcu_preempt11:50:030
3199190998227,15cyclictest3236764-21ssh10:17:470
3199190998038,4cyclictest16850irq/134-800000009:49:130
3199190997731,34cyclictest19350irq/34-mmc108:00:020
3199190997731,34cyclictest19350irq/34-mmc108:00:010
3199190997559,11cyclictest12-21ksoftirqd/009:20:020
3199190997527,16cyclictest3218853-21ssh09:38:590
3199190997467,3cyclictest0-21swapper/010:35:020
3199190997327,11cyclictest3214015-21ssh09:28:410
319919099727,38cyclictest16950irq/224-800000008:30:030
3199190997223,37cyclictest12-21ksoftirqd/010:45:020
3199190996927,15cyclictest3241998-21sshd10:29:150
3199190996926,12cyclictest3283195-21ssh11:57:110
3199190996861,3cyclictest0-21swapper/007:45:020
319919599675,19cyclictest131rcu_preempt11:40:031
3199190996623,11cyclictest3269016-21ssh11:26:450
319919099661,35cyclictest3234840-21ssh10:13:370
3199190996610,32cyclictest19350irq/34-mmc111:15:030
3199190996610,32cyclictest19350irq/34-mmc111:15:020
3199190996316,31cyclictest12-21ksoftirqd/011:50:510
3199199996241,7cyclictest3203859-21kworker/2:111:50:032
3199190996220,21cyclictest12-21ksoftirqd/011:45:010
3199190996017,22cyclictest3299823-21ssh12:32:280
3199190996017,22cyclictest3299823-21ssh12:32:280
3199190995926,5cyclictest131rcu_preempt12:25:050
3199199995850,5cyclictest3203859-21kworker/2:110:25:352
3199199995840,7cyclictest3228571-21kworker/2:011:23:572
319919099582,3cyclictest131rcu_preempt12:25:020
319920099579,5cyclictest131rcu_preempt11:21:443
319920099576,41cyclictest3235575-21ssh10:15:033
3199200995712,6cyclictest131rcu_preempt09:43:573
3199200995712,4cyclictest131rcu_preempt10:43:183
3199199995749,5cyclictest3203859-21kworker/2:110:42:332
3199199995748,5cyclictest3203859-21kworker/2:110:55:012
319919099573,4cyclictest131rcu_preempt07:50:020
3199190995728,26cyclictest12-21ksoftirqd/010:40:010
3199190995728,13cyclictest3229934-21ssh10:02:480
3199190995720,33cyclictest3200609-21latency_hist07:35:020
319920099567,10cyclictest37-21ksoftirqd/309:40:003
319920099562,12cyclictest37-21ksoftirqd/312:25:033
3199200995618,12cyclictest37-21ksoftirqd/309:00:013
3199190995628,12cyclictest12-21ksoftirqd/009:30:140
3199190995619,29cyclictest0-21swapper/012:06:460
319920099557,22cyclictest3200650-21kworker/u8:008:25:033
3199199995547,5cyclictest3272801-21kworker/2:312:29:072
3199199995546,5cyclictest3203859-21kworker/2:109:40:012
3199199995536,13cyclictest1-21systemd11:20:032
3199199995536,13cyclictest1-21systemd11:20:032
319919099556,33cyclictest19350irq/34-mmc108:20:030
3199190995547,5cyclictest3199197-21latency_hist07:10:010
3199190995533,7cyclictest3274315-21ssh11:38:140
3199190995511,23cyclictest3292303-21ssh12:16:250
319920099548,5cyclictest131rcu_preempt09:50:583
3199200995412,5cyclictest131rcu_preempt11:00:013
3199200995410,7cyclictest131rcu_preempt11:57:183
3199199995445,6cyclictest3203859-21kworker/2:109:35:002
3199190995427,12cyclictest3260266-21ssh11:08:010
3199190995427,12cyclictest3209207-21ssh09:18:100
3199190995426,13cyclictest3302217-21ssh12:37:430
3199190995426,13cyclictest3262734-21ssh11:13:140
3199190995424,15cyclictest3250979-21ssh10:48:150
3199190995423,12cyclictest0-21swapper/012:10:280
319920099539,5cyclictest131rcu_preempt09:47:493
319920099538,6cyclictest131rcu_preempt09:32:203
319920099536,10cyclictest37-21ksoftirqd/310:39:593
319920099533,7cyclictest37-21ksoftirqd/310:00:003
319920099531,8cyclictest3230884-21grep10:05:003
3199200995318,6cyclictest131rcu_preempt12:15:023
3199200995310,5cyclictest131rcu_preempt12:38:543
3199199995345,5cyclictest3203859-21kworker/2:109:24:132
3199199995344,5cyclictest3272801-21kworker/2:312:16:262
319919599539,5cyclictest131rcu_preempt09:20:021
3199195995335,12cyclictest275-21dbus-daemon10:00:011
3199190995326,12cyclictest3221612-21ssh09:44:590
3199190995325,13cyclictest3271787-21ssh11:32:490
319920099525,8cyclictest37-21ksoftirqd/307:10:013
3199200995242,7cyclictest37-21ksoftirqd/312:30:003
3199200995214,5cyclictest131rcu_preempt11:10:003
3199200995211,5cyclictest131rcu_preempt11:30:013
3199200995210,6cyclictest131rcu_preempt07:33:283
319919999526,36cyclictest3198910-21kworker/u8:207:25:012
3199199995245,4cyclictest3203859-21kworker/2:111:57:202
3199199995243,5cyclictest3203859-21kworker/2:109:14:192
3199195995235,13cyclictest1-21systemd07:55:001
3199190995213,19cyclictest12-21ksoftirqd/009:10:120
319920099511,7cyclictest3205374-21diskmemload11:44:593
319920099511,6cyclictest0-21swapper/310:50:003
319920099511,16cyclictest0-21swapper/309:05:013
319920099511,16cyclictest0-21swapper/309:05:013
3199200995111,5cyclictest131rcu_preempt09:14:253
3199199995143,5cyclictest3228571-21kworker/2:011:12:522
3199199995143,5cyclictest3203859-21kworker/2:111:54:592
3199199995143,5cyclictest3203859-21kworker/2:111:03:122
3199199995142,6cyclictest3203859-21kworker/2:109:43:522
3199199995142,5cyclictest3203859-21kworker/2:110:47:412
3199199995142,5cyclictest3203859-21kworker/2:109:16:012
3199190995135,11cyclictest309-21in:imuxsock10:05:020
319920099509,6cyclictest131rcu_preempt10:51:583
319920099509,5cyclictest131rcu_preempt11:19:573
319920099509,5cyclictest131rcu_preempt11:19:563
319920099508,6cyclictest131rcu_preempt11:50:013
319920099505,9cyclictest37-21ksoftirqd/310:32:573
319920099505,6cyclictest37-21ksoftirqd/311:35:023
319920099503,5cyclictest131rcu_preempt11:55:023
3199200995033,8cyclictest309-21in:imuxsock07:25:003
319920099503,10cyclictest37-21ksoftirqd/312:31:553
319920099503,10cyclictest37-21ksoftirqd/312:31:543
3199200995012,5cyclictest131rcu_preempt12:08:563
319920099501,21cyclictest0-21swapper/309:15:393
319920099501,11cyclictest37-21ksoftirqd/310:24:133
3199199995042,5cyclictest3203859-21kworker/2:109:59:182
3199199995037,9cyclictest1-21systemd11:45:012
319919999503,3cyclictest131rcu_preempt12:39:352
3199190995034,12cyclictest3467602-21systemd-journal08:56:260
3199190995020,13cyclictest3228578-21ssh10:00:020
319920099497,5cyclictest131rcu_preempt11:37:013
319920099497,5cyclictest131rcu_preempt10:08:083
319920099496,9cyclictest37-21ksoftirqd/312:20:013
319920099495,6cyclictest37-21ksoftirqd/310:28:013
3199200994941,5cyclictest37-21ksoftirqd/311:00:293
3199200994940,6cyclictest37-21ksoftirqd/310:20:023
3199200994929,13cyclictest3467602-21systemd-journal08:30:003
3199200994913,9cyclictest37-21ksoftirqd/311:10:393
319920099491,11cyclictest37-21ksoftirqd/309:23:243
3199200994910,12cyclictest37-21ksoftirqd/307:30:003
3199199994942,4cyclictest3203859-21kworker/2:111:06:362
3199199994941,5cyclictest3203859-21kworker/2:109:28:392
3199199994940,5cyclictest3203859-21kworker/2:112:11:582
3199199994940,5cyclictest3203859-21kworker/2:110:04:242
3199195994930,4cyclictest23-21ksoftirqd/111:15:031
3199190994938,8cyclictest12-21ksoftirqd/010:55:010
3199190994927,16cyclictest275-21dbus-daemon09:10:000
319920099489,6cyclictest131rcu_preempt09:10:013
319920099486,8cyclictest37-21ksoftirqd/308:40:013
319920099486,8cyclictest37-21ksoftirqd/308:40:003
3199200994814,6cyclictest131rcu_preempt12:02:033
3199199994842,3cyclictest3203859-21kworker/2:112:06:132
3199199994841,5cyclictest3203859-21kworker/2:110:31:232
3199199994840,5cyclictest3203859-21kworker/2:110:24:252
3199199994838,6cyclictest3228571-21kworker/2:011:29:262
3199195994827,16cyclictest275-21dbus-daemon11:35:011
3199195994811,6cyclictest131rcu_preempt12:20:001
3199190994826,17cyclictest16850irq/134-800000007:13:270
3199200994731,12cyclictest37-21ksoftirqd/308:50:003
3199200994730,11cyclictest3467602-21systemd-journal07:40:003
319920099471,16cyclictest0-21swapper/307:17:073
319920099471,16cyclictest0-21swapper/307:17:073
3199199994739,5cyclictest3228571-21kworker/2:011:32:112
3199199994739,5cyclictest3198776-21kworker/2:008:40:022
3199199994739,5cyclictest3198776-21kworker/2:008:40:012
3199199994738,6cyclictest3203859-21kworker/2:112:00:402
3199199994733,11cyclictest3201452-21sshd07:53:422
3199199994732,12cyclictest1-21systemd08:10:002
319919999472,9cyclictest3254477-21ssh10:55:232
319919599474,5cyclictest131rcu_preempt11:00:011
3199195994729,13cyclictest275-21dbus-daemon12:30:021
3199195994724,16cyclictest275-21dbus-daemon09:30:011
3199190994736,8cyclictest3240092-21cut10:25:020
3199190994722,10cyclictest3285830-21ssh12:02:460
319920099462,10cyclictest37-21ksoftirqd/308:20:003
3199200994619,6cyclictest131rcu_preempt09:25:403
3199199994639,4cyclictest3272801-21kworker/2:312:20:472
3199199994639,4cyclictest3203859-21kworker/2:111:38:202
3199199994638,5cyclictest3272801-21kworker/2:312:30:112
3199199994638,5cyclictest3272801-21kworker/2:312:30:102
3199199994638,5cyclictest3203859-21kworker/2:110:35:292
3199199994638,5cyclictest3203859-21kworker/2:110:19:352
3199199994637,5cyclictest3203859-21kworker/2:110:05:032
3199199994631,12cyclictest3467602-21systemd-journal07:36:262
319919599463,23cyclictest131rcu_preempt10:45:011
3199195994628,14cyclictest23-21ksoftirqd/109:25:021
3199195994623,5cyclictest23-21ksoftirqd/110:25:011
3199195994612,9cyclictest131rcu_preempt10:20:001
3199190994637,7cyclictest0-21swapper/011:05:020
3199190994618,13cyclictest3254743-21ssh10:56:070
3199199994538,5cyclictest3203859-21kworker/2:110:15:022
3199199994537,5cyclictest3203859-21kworker/2:109:50:442
3199199994532,10cyclictest1-21systemd08:55:012
319919599458,12cyclictest3284488-21sendmail12:00:001
3199195994529,11cyclictest1-21systemd12:15:011
3199195994523,18cyclictest3467602-21systemd-journal09:44:591
3199195994521,20cyclictest23-21ksoftirqd/112:01:411
3199190994518,11cyclictest3267114-21ssh11:22:330
319920099448,7cyclictest131rcu_preempt07:50:023
319920099444,10cyclictest37-21ksoftirqd/308:35:003
3199200994434,7cyclictest37-21ksoftirqd/307:55:023
3199200994433,8cyclictest361rcuc/308:05:003
3199200994430,11cyclictest275-21dbus-daemon07:15:013
3199200994428,12cyclictest3467602-21systemd-journal07:59:273
3199200994428,12cyclictest3467602-21systemd-journal07:59:263
3199199994435,7cyclictest30-21ksoftirqd/207:20:022
3199199994435,7cyclictest30-21ksoftirqd/207:20:022
3199199994429,10cyclictest3467602-21systemd-journal09:00:002
3199195994434,8cyclictest23-21ksoftirqd/110:10:531
3199195994430,8cyclictest1-21systemd11:55:011
3199195994429,12cyclictest1-21systemd11:30:001
3199195994421,5cyclictest23-21ksoftirqd/109:48:231
3199195994410,6cyclictest131rcu_preempt10:39:591
3199190994427,14cyclictest3199897-21sshd07:21:270
319920099439,6cyclictest131rcu_preempt08:55:003
319920099437,6cyclictest131rcu_preempt07:45:023
3199200994328,11cyclictest275-21dbus-daemon08:10:013
3199199994335,5cyclictest3203859-21kworker/2:109:45:062
3199199994329,11cyclictest3467602-21systemd-journal08:26:012
3199195994333,7cyclictest3267909-21sshd11:24:271
3199195994328,10cyclictest23-21ksoftirqd/108:55:011
3199195994310,5cyclictest131rcu_preempt12:10:031
3199190994336,4cyclictest16950irq/224-800000007:20:020
3199190994336,4cyclictest16950irq/224-800000007:20:010
319919099433,31cyclictest0-21swapper/008:25:020
3199199994233,5cyclictest3189435-21kworker/2:207:26:302
319919999421,3cyclictest131rcu_preempt08:25:022
319919599427,8cyclictest23-21ksoftirqd/110:55:011
3199195994236,3cyclictest231ksoftirqd/108:15:011
319919599423,5cyclictest131rcu_preempt10:30:001
3199195994233,6cyclictest3202157-21latency_hist08:05:011
3199195994232,7cyclictest3201691-21sshd07:56:281
3199195994232,7cyclictest3201691-21sshd07:56:271
3199195994231,7cyclictest0-21swapper/107:10:011
3199195994224,14cyclictest275-21dbus-daemon07:25:011
319919599421,9cyclictest0-21swapper/107:16:261
319919599421,9cyclictest0-21swapper/107:16:261
3199195994212,7cyclictest131rcu_preempt10:05:021
3199190994224,12cyclictest3202247-21modprobe08:10:010
319920099411,5cyclictest150-21pr/ttyS208:13:303
3199199994133,5cyclictest3198776-21kworker/2:008:40:272
319919599418,6cyclictest131rcu_preempt12:33:321
319919599418,6cyclictest131rcu_preempt12:33:311
319919599413,5cyclictest131rcu_preempt09:00:011
3199195994131,5cyclictest3244702-21munin-plugin-st10:35:011
3199195994127,11cyclictest23-21ksoftirqd/110:48:291
3199195994127,10cyclictest23-21ksoftirqd/109:34:591
3199195994126,11cyclictest23-21ksoftirqd/110:07:001
3199195994123,14cyclictest3265820-21munin-run11:20:031
3199195994123,14cyclictest3265820-21munin-run11:20:031
3199200994014,5cyclictest131rcu_preempt08:45:023
3199199994032,5cyclictest3203859-21kworker/2:109:05:022
3199199994032,5cyclictest3203859-21kworker/2:109:05:012
3199195994029,8cyclictest23-21ksoftirqd/112:20:531
3199195994027,10cyclictest23-21ksoftirqd/111:06:271
3199195994027,10cyclictest23-21ksoftirqd/109:51:351
3199195994025,10cyclictest23-21ksoftirqd/111:40:111
3199195994023,4cyclictest23-21ksoftirqd/108:10:011
3199195994017,19cyclictest23-21ksoftirqd/111:00:381
3199199993930,5cyclictest3189435-21kworker/2:207:12:272
3199199993927,9cyclictest3205344-21sshd09:07:262
319919599397,8cyclictest23-21ksoftirqd/107:31:281
3199195993928,8cyclictest3200149-21sshd07:26:261
3199195993925,10cyclictest309-21in:imuxsock08:35:031
3199195993925,10cyclictest309-21in:imuxsock08:35:021
3199195993916,16cyclictest275-21dbus-daemon08:25:011
319919599391,4cyclictest131rcu_preempt11:50:011
3199190993931,5cyclictest3203805-21sh08:40:000
3199190993931,5cyclictest3203805-21sh08:40:000
319919099391,9cyclictest3204703-21latency_hist08:55:030
3199190993919,14cyclictest275-21dbus-daemon08:15:010
319919099391,34cyclictest3200720-21systemd-run07:40:000
3199199993830,5cyclictest3198776-21kworker/2:007:40:522
319919999381,3cyclictest131rcu_preempt08:35:022
319919599385,6cyclictest131rcu_preempt07:15:011
3199195993820,14cyclictest23-21ksoftirqd/107:40:001
3199190993828,7cyclictest12-21ksoftirqd/009:05:030
3199190993828,7cyclictest12-21ksoftirqd/009:05:020
3199190993819,13cyclictest275-21dbus-daemon08:50:000
3199199993723,11cyclictest30-21ksoftirqd/207:59:262
3199199993723,11cyclictest30-21ksoftirqd/207:59:262
3199195993711,6cyclictest131rcu_preempt08:35:011
319919599371,11cyclictest23-21ksoftirqd/109:15:031
3199199993628,5cyclictest3198776-21kworker/2:008:15:012
3199199993628,5cyclictest3198776-21kworker/2:008:05:022
3199199993621,11cyclictest3202780-21systemctl08:20:012
319919599367,5cyclictest131rcu_preempt09:10:001
319919599364,7cyclictest131rcu_preempt08:50:031
3199195993623,11cyclictest23-21ksoftirqd/112:38:111
3199195993615,5cyclictest131rcu_preempt07:46:281
3199190993626,7cyclictest3200946-21sshd07:43:410
3199199993527,5cyclictest3198776-21kworker/2:007:30:532
3199199993524,8cyclictest3201197-21sshd07:48:402
319919999351,3cyclictest131rcu_preempt08:50:022
319919599354,5cyclictest131rcu_preempt08:20:001
319919599351,7cyclictest0-21swapper/109:05:031
319919599351,7cyclictest0-21swapper/109:05:021
3199195993516,5cyclictest131rcu_preempt08:30:001
319919099352,4cyclictest3201965-21modprobe08:00:020
319919599342,6cyclictest131rcu_preempt08:42:271
3199190993421,10cyclictest3203277-21sshd08:28:270
319919599322,5cyclictest131rcu_preempt07:45:001
319919099321,29cyclictest0-21swapper/008:43:480
319919999263,3cyclictest131rcu_preempt07:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional