You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-22 - 07:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Mar 22, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
540899455,25cyclictest28366irq/58-eth022:50:133
5408994530,9cyclictest28366irq/58-eth022:35:153
540899452,26cyclictest28366irq/58-eth022:40:123
540899447,19cyclictest28366irq/58-eth021:15:113
5408994324,9cyclictest28366irq/58-eth000:30:113
5384994311,3cyclictest131rcu_preempt22:55:120
5384994311,3cyclictest131rcu_preempt22:55:110
540899423,24cyclictest28366irq/58-eth023:50:133
540899394,21cyclictest28366irq/58-eth019:20:123
540899392,20cyclictest28366irq/58-eth000:20:133
5408993921,14cyclictest28366irq/58-eth000:25:133
5408993910,18cyclictest0-21swapper/321:40:123
538499375,10cyclictest12-21ksoftirqd/000:25:140
539299361,3cyclictest131rcu_preempt23:35:131
539299361,22cyclictest3560-21ssh23:40:131
540899344,16cyclictest28366irq/58-eth023:10:123
540899343,12cyclictest0-21swapper/323:55:133
5408993316,12cyclictest28366irq/58-eth023:00:123
539299338,21cyclictest21-21ksoftirqd/100:00:121
539299335,17cyclictest21-21ksoftirqd/119:40:131
539299320,3cyclictest131rcu_preempt21:15:111
540899313,17cyclictest28366irq/58-eth022:30:143
539299316,16cyclictest21-21ksoftirqd/121:30:101
5392993114,5cyclictest23660-21diskmemload22:10:191
5400993011,9cyclictest28437-21ssh23:00:112
5392993015,5cyclictest22892-21munin-run23:15:011
5408992918,7cyclictest0-21swapper/322:25:143
5400992911,3cyclictest27-21ksoftirqd/223:42:052
5400992910,10cyclictest0-21swapper/200:21:382
539299299,5cyclictest21-21ksoftirqd/122:30:011
539299291,18cyclictest17402-21ssh22:00:141
540099288,14cyclictest0-21swapper/221:30:042
5400992818,4cyclictest27-21ksoftirqd/223:35:122
5400992818,4cyclictest27-21ksoftirqd/222:28:312
5400992810,3cyclictest27-21ksoftirqd/223:14:552
539299281,21cyclictest12408-21ssh00:37:571
5384992814,3cyclictest12-21ksoftirqd/022:35:210
540899279,12cyclictest28366irq/58-eth021:20:153
5408992711,10cyclictest0-21swapper/323:38:133
540099276,4cyclictest4973-21/usr/sbin/munin22:45:222
5400992716,5cyclictest27-21ksoftirqd/200:00:112
5400992713,10cyclictest27-21ksoftirqd/222:20:132
5400992710,3cyclictest27-21ksoftirqd/220:30:212
5392992717,4cyclictest0-21swapper/100:33:211
5384992714,3cyclictest12-21ksoftirqd/023:27:120
5384992714,3cyclictest12-21ksoftirqd/022:25:300
5384992712,3cyclictest12-21ksoftirqd/022:30:470
5384992711,3cyclictest12-21ksoftirqd/023:30:250
540899263,15cyclictest0-21swapper/323:43:143
540099269,7cyclictest27-21ksoftirqd/223:45:032
540099267,3cyclictest27-21ksoftirqd/200:30:152
540099265,15cyclictest0-21swapper/222:07:342
540099265,15cyclictest0-21swapper/221:22:182
540099264,14cyclictest0-21swapper/222:54:172
5400992617,4cyclictest27-21ksoftirqd/223:15:092
5400992616,5cyclictest27-21ksoftirqd/221:43:412
540099261,10cyclictest9582-21irqrtprio23:25:182
5392992617,3cyclictest25472-21ssh00:26:201
5392992615,5cyclictest11484-21kworker/1:221:38:021
5392992615,5cyclictest11484-21kworker/1:221:38:011
5384992613,3cyclictest12-21ksoftirqd/021:58:470
5384992613,3cyclictest12-21ksoftirqd/000:30:110
5384992613,3cyclictest12-21ksoftirqd/000:15:310
540899252,15cyclictest28366irq/58-eth023:15:123
540899252,13cyclictest28366irq/58-eth023:25:123
540899252,12cyclictest28366irq/58-eth022:55:113
540899252,12cyclictest28366irq/58-eth022:55:113
5408992512,6cyclictest28466irq/59-eth021:25:133
540099257,3cyclictest27-21ksoftirqd/221:10:232
540099257,3cyclictest27-21ksoftirqd/221:10:222
540099255,4cyclictest0-21swapper/223:32:392
540099251,6cyclictest1245566ssh23:08:262
5400992516,4cyclictest27-21ksoftirqd/221:27:292
5400992512,3cyclictest27-21ksoftirqd/222:30:212
539299259,8cyclictest21-21ksoftirqd/100:13:591
539299253,14cyclictest0-21swapper/122:20:241
5392992515,3cyclictest24229-21cut23:15:191
5392992511,7cyclictest23660-21diskmemload22:56:381
5392992511,7cyclictest23660-21diskmemload22:56:371
5384992514,3cyclictest12-21ksoftirqd/021:42:350
5384992513,3cyclictest12-21ksoftirqd/021:50:230
5384992513,3cyclictest12-21ksoftirqd/021:20:130
5384992513,3cyclictest12-21ksoftirqd/000:35:220
5384992512,3cyclictest12-21ksoftirqd/022:02:350
5384992512,3cyclictest12-21ksoftirqd/021:10:020
5384992512,3cyclictest12-21ksoftirqd/021:10:020
5384992510,3cyclictest131rcu_preempt21:15:120
540899241,12cyclictest28366irq/58-eth019:40:123
540099248,11cyclictest27-21ksoftirqd/222:40:422
540099245,7cyclictest16334-21ssh22:35:202
540099245,6cyclictest13546-21ssh21:55:562
540099243,7cyclictest131rcu_preempt00:06:072
540099241,7cyclictest2784966ssh00:29:102
540099241,6cyclictest89666ssh21:15:162
5400992415,4cyclictest27-21ksoftirqd/200:17:132
5400992411,8cyclictest0-21swapper/221:38:052
5400992411,8cyclictest0-21swapper/221:38:052
5400992410,9cyclictest0-21swapper/200:11:422
540099240,13cyclictest925566ssh22:12:112
539299243,15cyclictest3089-21ssh21:50:271
5392992415,3cyclictest14907-21kworker/1:100:06:161
539299241,15cyclictest11484-21kworker/1:221:40:231
538499249,3cyclictest12-21ksoftirqd/020:55:160
538499249,3cyclictest12-21ksoftirqd/020:55:150
5384992413,3cyclictest12-21ksoftirqd/023:35:200
5384992412,3cyclictest12-21ksoftirqd/022:10:130
5384992411,3cyclictest12-21ksoftirqd/021:35:160
5384992411,3cyclictest12-21ksoftirqd/021:35:150
540899233,13cyclictest0-21swapper/322:00:143
540899233,13cyclictest0-21swapper/300:37:573
540899232,10cyclictest0-21swapper/322:20:143
5408992313,5cyclictest0-21swapper/321:55:133
540099239,5cyclictest0-21swapper/222:16:192
540099236,12cyclictest27-21ksoftirqd/220:50:132
540099236,12cyclictest27-21ksoftirqd/220:50:132
540099235,8cyclictest14274-21/usr/sbin/munin21:00:172
540099233,2cyclictest131rcu_preempt21:46:532
540099232,6cyclictest261rcuc/222:00:142
540099232,2cyclictest131rcu_preempt23:58:482
540099231,7cyclictest131rcu_preempt23:50:242
540099231,6cyclictest698766ssh21:53:542
540099231,6cyclictest557766ssh23:21:212
540099231,6cyclictest3090366/usr/sbin/munin20:10:232
5400992311,9cyclictest24766-21ssh22:56:122
5400992311,9cyclictest24766-21ssh22:56:112
5400992310,8cyclictest27-21ksoftirqd/200:37:012
5400992310,3cyclictest27-21ksoftirqd/220:40:252
5400992310,2cyclictest27-21ksoftirqd/220:20:192
539299237,6cyclictest21-21ksoftirqd/121:25:131
539299234,12cyclictest0-21swapper/120:20:201
539299233,14cyclictest2869-21ssh23:02:511
539299233,14cyclictest21784-21kthreadcore19:25:231
539299232,15cyclictest633-21Async12:11:531
539299231,2cyclictest131rcu_preempt23:50:471
539299231,2cyclictest131rcu_preempt22:41:111
539299231,2cyclictest131rcu_preempt19:20:281
5392992311,9cyclictest21-21ksoftirqd/121:24:311
539299231,16cyclictest9015-21kworker/1:220:55:221
539299231,16cyclictest9015-21kworker/1:220:55:211
539299231,16cyclictest0-21swapper/121:56:051
539299231,14cyclictest3023-21kworker/1:220:15:281
539299231,14cyclictest0-21swapper/121:05:241
538499238,7cyclictest12-21ksoftirqd/000:13:550
5384992314,3cyclictest12-21ksoftirqd/020:45:260
5384992312,3cyclictest12-21ksoftirqd/023:40:320
5384992312,3cyclictest12-21ksoftirqd/020:40:250
5384992311,3cyclictest12-21ksoftirqd/022:40:130
5384992310,3cyclictest12-21ksoftirqd/023:59:310
5384992310,3cyclictest12-21ksoftirqd/019:30:010
540899223,15cyclictest0-21swapper/300:00:143
540899222,8cyclictest0-21swapper/321:35:113
540899222,8cyclictest0-21swapper/321:35:113
540099226,7cyclictest27-21ksoftirqd/219:50:262
540099221,6cyclictest2960666/usr/sbin/munin19:35:262
5400992212,8cyclictest0-21swapper/220:00:322
539299227,9cyclictest11075-21irqstats22:50:191
539299225,7cyclictest131rcu_preempt21:46:181
539299224,9cyclictest21-21ksoftirqd/122:15:181
539299223,13cyclictest11640-21kworker/1:219:55:211
539299221,2cyclictest131rcu_preempt23:31:171
539299221,2cyclictest131rcu_preempt23:29:321
538499229,3cyclictest12-21ksoftirqd/019:35:160
5384992211,3cyclictest12-21ksoftirqd/021:48:150
5384992211,3cyclictest12-21ksoftirqd/000:23:330
5384992210,3cyclictest12-21ksoftirqd/023:50:190
5384992210,3cyclictest12-21ksoftirqd/023:24:580
5384992210,3cyclictest12-21ksoftirqd/021:30:340
5384992210,3cyclictest12-21ksoftirqd/000:02:300
5384992210,2cyclictest12-21ksoftirqd/019:45:020
540899213,13cyclictest0-21swapper/322:10:153
540899211,15cyclictest0-21swapper/321:30:113
540099219,8cyclictest27-21ksoftirqd/219:45:252
540099219,7cyclictest27-21ksoftirqd/220:35:182
540099219,3cyclictest27-21ksoftirqd/219:10:262
540099219,2cyclictest27-21ksoftirqd/219:25:292
540099218,9cyclictest0-21swapper/220:15:172
540099218,8cyclictest27-21ksoftirqd/220:55:272
540099218,8cyclictest27-21ksoftirqd/220:55:272
540099216,7cyclictest131rcu_preempt20:09:092
540099211,3cyclictest131rcu_preempt19:30:222
5400992110,2cyclictest27-21ksoftirqd/219:20:172
540099210,7cyclictest91-21pr/ttyS020:00:012
539299219,2cyclictest21-21ksoftirqd/122:39:331
539299212,11cyclictest21-21ksoftirqd/121:11:121
539299212,11cyclictest21-21ksoftirqd/121:11:111
539299211,2cyclictest131rcu_preempt23:46:101
539299211,11cyclictest30696-21ntp_192.168.11420:40:201
539299211,11cyclictest0-21swapper/120:50:131
539299211,11cyclictest0-21swapper/120:50:131
539299210,2cyclictest131rcu_preempt22:49:231
538499219,3cyclictest12-21ksoftirqd/023:48:440
538499219,3cyclictest12-21ksoftirqd/023:05:080
538499219,3cyclictest12-21ksoftirqd/020:15:010
538499218,3cyclictest12-21ksoftirqd/021:05:230
538499212,12cyclictest12-21ksoftirqd/020:15:130
5384992112,3cyclictest12-21ksoftirqd/022:07:470
5384992112,2cyclictest12-21ksoftirqd/019:10:260
5384992111,3cyclictest12-21ksoftirqd/020:20:170
5384992111,2cyclictest12-21ksoftirqd/023:10:160
5384992110,3cyclictest12-21ksoftirqd/019:55:110
5384992110,2cyclictest12-21ksoftirqd/021:25:140
5384992110,2cyclictest12-21ksoftirqd/000:05:240
540899204,10cyclictest0-21swapper/321:10:143
540899204,10cyclictest0-21swapper/321:10:133
540899203,12cyclictest0-21swapper/321:45:153
540099209,7cyclictest27-21ksoftirqd/220:46:572
540099207,2cyclictest27-21ksoftirqd/220:25:142
540099202,6cyclictest131rcu_preempt19:40:122
540099200,3cyclictest131rcu_preempt21:05:202
539299201,9cyclictest0-21swapper/119:30:281
539299201,3cyclictest0-21swapper/122:33:541
539299201,2cyclictest131rcu_preempt00:18:551
539299201,2cyclictest131rcu_preempt00:00:011
539299201,13cyclictest23660-21diskmemload23:05:101
539299201,12cyclictest0-21swapper/119:15:121
539299201,12cyclictest0-21swapper/119:15:111
539299201,11cyclictest11057-21df_inode00:20:151
539299201,11cyclictest0-21swapper/122:05:521
539299201,10cyclictest0-21swapper/120:45:131
539299201,10cyclictest0-21swapper/120:10:201
5392992010,3cyclictest0-21swapper/123:21:411
538499209,2cyclictest12-21ksoftirqd/022:22:250
5384992011,2cyclictest12-21ksoftirqd/020:25:250
540899199,8cyclictest0-21swapper/322:48:503
540899199,7cyclictest0-21swapper/323:20:263
540899198,8cyclictest0-21swapper/323:46:063
540899196,10cyclictest0-21swapper/300:05:193
540899192,11cyclictest0-21swapper/320:25:143
540899192,10cyclictest0-21swapper/319:25:143
540899191,12cyclictest0-21swapper/323:30:163
540899191,12cyclictest0-21swapper/320:05:313
540899191,11cyclictest0-21swapper/319:35:233
5408991910,6cyclictest0-21swapper/300:13:513
540099198,3cyclictest27-21ksoftirqd/219:15:152
540099198,3cyclictest27-21ksoftirqd/219:15:142
539299193,10cyclictest27712-21grep20:05:221
539299191,9cyclictest0-21swapper/120:30:291
539299191,9cyclictest0-21swapper/119:50:271
539299191,12cyclictest0-21swapper/120:25:141
539299190,3cyclictest10668-21unixbench_multi19:45:291
538499199,3cyclictest12-21ksoftirqd/019:45:290
538499192,2cyclictest131rcu_preempt22:53:160
5384991913,4cyclictest28448-21sh23:00:110
5384991913,3cyclictest12-21ksoftirqd/022:45:280
5384991912,5cyclictest14865-21cron19:20:000
5384991912,5cyclictest14865-21cron19:20:000
540899183,11cyclictest0-21swapper/322:15:163
540899182,10cyclictest0-21swapper/320:40:123
540899182,10cyclictest0-21swapper/320:30:143
540899182,10cyclictest0-21swapper/320:20:123
540899182,10cyclictest0-21swapper/319:55:143
540899181,13cyclictest0-21swapper/320:50:153
540899181,13cyclictest0-21swapper/320:50:143
540899181,10cyclictest0-21swapper/323:05:123
539299182,8cyclictest21-21ksoftirqd/121:00:271
539299181,3cyclictest131rcu_preempt19:35:201
539299180,10cyclictest20980-21apt-get20:00:031
5384991813,3cyclictest12-21ksoftirqd/021:00:270
5384991813,3cyclictest0-21swapper/022:16:540
5384991812,4cyclictest0-21swapper/023:15:390
5384991812,3cyclictest27095-21lockfile-create20:40:010
5384991812,3cyclictest12-21ksoftirqd/020:05:250
538499180,2cyclictest131rcu_preempt19:30:270
540899179,6cyclictest0-21swapper/322:05:213
540899178,6cyclictest0-21swapper/321:52:213
540899172,9cyclictest0-21swapper/321:05:153
540899172,11cyclictest0-21swapper/300:15:143
540899172,10cyclictest0-21swapper/321:00:043
539299171,10cyclictest0-21swapper/120:35:121
538499176,8cyclictest19682-21cron19:25:010
5384991712,3cyclictest12-21ksoftirqd/020:00:200
5384991711,3cyclictest12-21ksoftirqd/020:55:010
5384991711,3cyclictest12-21ksoftirqd/020:55:000
5384991711,3cyclictest12-21ksoftirqd/020:30:240
540899162,10cyclictest0-21swapper/319:15:133
540899162,10cyclictest0-21swapper/319:15:133
540899161,9cyclictest0-21swapper/319:45:223
5384991610,3cyclictest12-21ksoftirqd/019:50:190
540899152,8cyclictest0-21swapper/319:30:053
539299151,2cyclictest131rcu_preempt19:10:011
540899149,3cyclictest0-21swapper/320:10:153
540899142,7cyclictest0-21swapper/320:15:133
540899138,3cyclictest0-21swapper/320:00:243
540899131,6cyclictest0-21swapper/319:50:133
540899131,10cyclictest0-21swapper/321:00:013
540899131,10cyclictest0-21swapper/321:00:013
538499131,9cyclictest5262-21latency_hist19:10:010
540899121,9cyclictest0-21swapper/320:35:193
540899112,5cyclictest0-21swapper/320:45:123
540899111,7cyclictest0-21swapper/319:10:173
540099116,3cyclictest5451-21latency_hist19:10:012
540899101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional