You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-19 - 04:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Aug 19, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2954729551,39sleep00-21swapper/019:09:590
2950129273,14sleep10-21swapper/119:09:231
2953328658,9sleep30-21swapper/319:09:503
2936128270,8sleep20-21swapper/219:07:362
174632440,0sleep20-21swapper/200:39:022
2962399251,7cyclictest13114-21ssh23:00:150
2963399240,23cyclictest0-21swapper/200:32:532
2963799236,16cyclictest10937-21kworker/u8:023:38:343
29637992310,12cyclictest32252-21apt-get19:15:143
2963799230,3cyclictest0-21swapper/300:10:153
2963799230,22cyclictest0-21swapper/300:38:043
29637992221,1cyclictest0-21swapper/322:05:153
2963799220,18cyclictest0-21swapper/321:52:553
2962899222,20cyclictest0-21swapper/123:15:341
2962899222,20cyclictest0-21swapper/123:15:331
29628992221,1cyclictest0-21swapper/100:38:041
2963799216,10cyclictest9707-21/usr/sbin/munin00:15:163
2962899210,16cyclictest0-21swapper/121:49:051
2962899210,16cyclictest0-21swapper/121:49:051
2962399212,3cyclictest141rcu_preempt22:43:120
2963399200,20cyclictest0-21swapper/223:11:252
2963399200,20cyclictest0-21swapper/222:56:382
2963399200,20cyclictest0-21swapper/221:51:312
29623992019,1cyclictest22212-21diskmemload23:46:290
29623992019,1cyclictest0-21swapper/020:55:140
29633991914,1cyclictest32-21ksoftirqd/223:54:422
2963399190,17cyclictest0-21swapper/222:17:412
29628991915,3cyclictest0-21swapper/121:36:281
29637991817,1cyclictest0-21swapper/322:10:373
2962899181,2cyclictest0-21swapper/122:52:041
2963799170,1cyclictest0-21swapper/323:49:213
2963799170,16cyclictest0-21swapper/322:38:093
2963399170,17cyclictest0-21swapper/222:09:162
2963399170,16cyclictest0-21swapper/221:22:362
29628991715,2cyclictest0-21swapper/123:32:341
2962899170,2cyclictest0-21swapper/120:15:151
2962899170,17cyclictest0-21swapper/100:07:491
2962399171,2cyclictest0-21swapper/021:27:080
29637991614,1cyclictest0-21swapper/322:25:353
2963799160,16cyclictest0-21swapper/322:15:193
2963399160,16cyclictest0-21swapper/200:12:152
2963399160,15cyclictest0-21swapper/223:39:192
2963399160,15cyclictest0-21swapper/223:16:372
2963399160,15cyclictest0-21swapper/223:16:372
2962899169,6cyclictest15374-21apache223:50:171
2962899162,1cyclictest0-21swapper/123:39:381
13132160,0chrt0-21swapper/100:29:311
2963799154,1cyclictest141rcu_preempt22:48:523
29637991514,1cyclictest0-21swapper/323:29:513
29637991511,4cyclictest0-21swapper/323:52:293
2963799150,14cyclictest15778-21sensors23:20:223
2963799150,12cyclictest0-21swapper/323:30:083
2963399150,5cyclictest0-21swapper/222:32:472
2963399150,3cyclictest0-21swapper/220:05:192
2963399150,14cyclictest0-21swapper/222:10:092
2963399150,0cyclictest0-21swapper/222:40:132
29628991514,1cyclictest0-21swapper/122:15:191
29628991510,1cyclictest0-21swapper/122:39:461
2962899150,5cyclictest0-21swapper/123:56:121
2962899150,4cyclictest31926-21rm21:54:311
2962899150,14cyclictest0-21swapper/121:18:271
203932150,0chrt0-21swapper/300:20:403
2963799148,5cyclictest39-21ksoftirqd/320:10:183
2963799140,1cyclictest0-21swapper/321:22:443
2963799140,14cyclictest0-21swapper/300:03:043
2963399149,3cyclictest32-21ksoftirqd/221:15:112
2963399140,1cyclictest0-21swapper/221:11:572
2963399140,14cyclictest0-21swapper/200:16:362
2962899148,2cyclictest10660-21sh22:00:241
2962899147,6cyclictest23888-21sh23:45:141
29628991411,2cyclictest19612-21cron22:45:001
2962899140,14cyclictest0-21swapper/119:40:291
2962399146,7cyclictest16285-21munin-run19:50:000
2962399142,9cyclictest0-21swapper/022:08:280
29623991410,3cyclictest32402-21ntp_states21:15:210
2962399140,14cyclictest0-21swapper/000:32:530
2963799135,2cyclictest225-21systemd-journal19:22:133
2963799133,6cyclictest10663-21sensors22:00:243
29637991310,2cyclictest12048-21tr22:40:143
2963399135,7cyclictest31988-21systemd00:09:062
2963399131,1cyclictest22212-21diskmemload22:46:042
29633991310,2cyclictest5492-21latency_hist20:35:012
29633991310,2cyclictest29113-21grep20:15:182
29633991310,2cyclictest10399-21wc19:35:202
2963399130,1cyclictest0-21swapper/221:25:322
2963399130,12cyclictest26084-21cat00:25:002
2963399130,12cyclictest0-21swapper/223:56:242
29628991310,2cyclictest22212-21diskmemload22:32:101
2962899130,13cyclictest0-21swapper/123:06:461
2962899130,13cyclictest0-21swapper/100:15:321
2962399134,8cyclictest29857-21sshd23:10:130
29623991310,2cyclictest2980-21grep19:20:180
29623991310,2cyclictest25669-21cat21:50:170
29623991310,2cyclictest23791-21ssh23:05:410
2962399130,10cyclictest0-21swapper/020:05:190
2963799129,2cyclictest9683-21grep23:55:223
2963799129,2cyclictest5782-21ssh21:57:533
2963799129,2cyclictest30342-21ssh22:50:243
2963799129,2cyclictest23786-21ssh21:10:193
2963799129,2cyclictest23524-21rm23:05:283
2963799129,2cyclictest21670-21ssh21:48:003
2963799129,2cyclictest21670-21ssh21:48:003
2963799129,2cyclictest21112-21ssh23:05:003
2963799129,2cyclictest20006-21rm21:27:283
2963799126,2cyclictest225-21systemd-journal19:35:003
2963799124,2cyclictest225-21systemd-journal20:08:133
2963799122,8cyclictest31074-21apt-get21:15:133
29637991211,1cyclictest55650irq/127-enp2s0-00:30:143
29637991210,1cyclictest7006-21grep23:15:203
29637991210,1cyclictest7006-21grep23:15:203
29637991210,1cyclictest30083-21grep20:15:283
2963799120,4cyclictest225-21systemd-journal21:39:153
2963799120,11cyclictest9890-21grep19:35:163
2963799120,11cyclictest12638-21cat23:00:003
2963399129,2cyclictest31283-21wc19:10:262
2963399129,2cyclictest21573-21tr20:00:152
2963399129,2cyclictest18199-21grep19:50:282
2963399129,2cyclictest17438-21apt21:00:002
2963399129,2cyclictest17313-21cut21:45:192
2963399129,2cyclictest17313-21cut21:45:192
2963399129,2cyclictest14981-21sed23:20:162
2963399129,2cyclictest13455-21chrt21:43:072
2963399129,2cyclictest13455-21chrt21:43:072
2963399129,2cyclictest13344-21sed19:40:292
2963399127,2cyclictest311rcuc/200:00:272
2963399126,1cyclictest22212-21diskmemload23:29:112
2963399123,7cyclictest16898-21kworker/2:121:30:262
29633991210,1cyclictest0-21swapper/222:50:382
2963399120,1cyclictest0-21swapper/221:05:142
2963399120,12cyclictest0-21swapper/223:30:352
2962899129,3cyclictest25875-21rm21:11:191
2962899129,2cyclictest8371-21grep20:40:161
2962899129,2cyclictest5493-21apt20:35:011
2962899129,2cyclictest3831-21sed19:20:271
2962899129,2cyclictest32544-21cat20:20:261
2962899129,2cyclictest32126-21cat19:15:001
2962899129,2cyclictest24686-21chrt23:25:321
2962899129,2cyclictest17699-21tail00:00:181
2962899129,2cyclictest16763-21ssh21:25:221
2962899129,2cyclictest15721-21expr20:55:161
2962899129,2cyclictest1486-21expr20:25:161
2962899129,2cyclictest14567-21cat19:45:171
2962899129,2cyclictest14018-21ssh23:20:071
2962899129,2cyclictest1378-21aten2_r7power_e00:10:161
2962899129,2cyclictest10050-21tail19:35:181
2962899128,3cyclictest24271-21cat21:30:161
2962899128,1cyclictest38150irq/131-snd_hda23:13:581
2962899127,5cyclictest15720-21kworker/1:100:30:201
2962899123,2cyclictest25937-21awk00:25:001
29628991211,1cyclictest38150irq/131-snd_hda21:42:091
29628991211,1cyclictest38150irq/131-snd_hda21:42:091
29628991210,1cyclictest17780-21grep22:05:161
2962899120,1cyclictest0-21swapper/121:20:251
2962899120,12cyclictest0-21swapper/122:10:371
2962399129,2cyclictest8912-21ssh21:20:300
2962399129,2cyclictest8802-21cat23:35:260
2962399129,2cyclictest5750-21uname22:55:190
2962399129,2cyclictest31059-21chrt22:31:530
2962399129,2cyclictest19348-21grep21:00:260
2962399129,2cyclictest11014-21grep22:00:280
2962399129,2cyclictest1038-21grep19:15:230
2962399127,4cyclictest22764-21ssh21:10:140
2962399125,5cyclictest22212-21diskmemload00:02:270
2962399121,4cyclictest0-21swapper/000:05:230
2962399121,1cyclictest0-21swapper/023:42:310
29623991210,1cyclictest8096-21tr21:40:130
29623991210,1cyclictest8096-21tr21:40:130
2962399120,9cyclictest0-21swapper/021:48:290
2962399120,9cyclictest0-21swapper/021:48:280
2962399120,3cyclictest0-21swapper/023:56:520
2962399120,3cyclictest0-21swapper/023:17:270
2962399120,3cyclictest0-21swapper/023:17:270
2962399120,12cyclictest0-21swapper/022:52:020
2962399120,10cyclictest0-21swapper/022:18:380
2963799119,1cyclictest16989-21cat19:50:163
2963799119,1cyclictest13092-21mailstats19:40:263
2963799118,2cyclictest560-21chrt20:22:583
2963799118,2cyclictest2951-21chrt20:28:023
2963799118,2cyclictest28744-21cat22:30:223
2963799118,2cyclictest25675-21sed21:30:273
2963799118,2cyclictest23385-21taskset20:04:423
2963799118,2cyclictest21753-21grep21:05:273
2963799118,2cyclictest19309-21expr19:55:173
2963799118,2cyclictest19309-21expr19:55:173
2963799118,2cyclictest14214-21grep19:45:163
2963799118,2cyclictest13390-21sh22:21:233
2963799118,2cyclictest11233-21tail20:45:163
2963799118,2cyclictest10356-21ssh21:40:403
2963799118,2cyclictest10356-21ssh21:40:403
2963799117,3cyclictest5669-21date20:35:013
2963799117,3cyclictest30598-21grep19:10:173
2963799117,3cyclictest27193-21grep00:05:313
2963799112,6cyclictest26421-21/usr/sbin/munin00:25:273
2963799110,9cyclictest0-21swapper/323:40:373
2963799110,9cyclictest0-21swapper/321:01:143
2963799110,10cyclictest30599-21sh23:10:183
2963399119,1cyclictest32-21ksoftirqd/222:03:322
2963399119,1cyclictest27299-21grep00:25:182
2963399119,1cyclictest15813-21tr23:40:142
2963399118,2cyclictest8619-21expr20:40:162
2963399118,2cyclictest6309-21sh22:36:102
2963399118,2cyclictest6034-21tr20:35:162
2963399118,2cyclictest3348-21sh21:36:442
2963399118,2cyclictest30926-21sh23:49:232
2963399118,2cyclictest2629-21expr19:20:162
2963399118,2cyclictest22745-21awk23:05:202
2963399118,2cyclictest20448-21ssh22:25:242
2963399118,2cyclictest2028-21wc19:20:002
2963399118,2cyclictest16464-21sh23:01:302
2963399118,2cyclictest11528-21ssh22:20:232
2963399110,10cyclictest4560-21ssh21:56:562
2963399110,10cyclictest2269-21grep20:25:222
2962899119,2cyclictest20584-21ssh23:04:431
2962899119,1cyclictest20108-21ssh22:45:021
2962899119,1cyclictest177282sleep121:00:141
2962899119,1cyclictest16548-21cat23:40:181
2962899119,1cyclictest12404-21ssh22:20:361
2962899118,2cyclictest7200-21tr19:30:161
2962899118,2cyclictest6897-21cat19:30:001
2962899118,2cyclictest6085-21cat20:35:161
2962899118,2cyclictest4627-21ssh22:55:101
2962899118,2cyclictest17303-21tail19:50:181
2962899118,1cyclictest22473-21ntpq20:00:221
2962899116,2cyclictest0-21swapper/119:15:141
2962899114,4cyclictest225-21systemd-journal20:50:001
2962899114,4cyclictest18085-21cron22:25:021
2962899112,8cyclictest668-21apache_volume21:55:131
2962399119,1cyclictest9620-21grep20:40:270
2962399119,1cyclictest4279-21tail22:35:190
2962399119,1cyclictest31984-21munin-run19:15:000
2962399119,1cyclictest25215-21tr21:30:220
2962399119,1cyclictest18180-21grep19:50:280
2962399119,1cyclictest12209-21grep20:45:260
2962399118,2cyclictest8089-21cut20:39:590
2962399118,2cyclictest3366-21ssh21:36:450
2962399118,2cyclictest28038-21ssh22:49:520
2962399118,2cyclictest27712-21needreboot00:25:210
2962399118,2cyclictest25825-21ssh22:10:120
2962399118,2cyclictest22034-21head20:00:180
2962399118,2cyclictest15952-21cat23:20:240
2962399118,2cyclictest12561-21ssh00:16:050
2962399118,2cyclictest10316-21wget22:20:160
2962399116,5cyclictest15374-21apache219:35:210
2962399116,1cyclictest22212-21diskmemload00:10:210
2962399113,4cyclictest225-21systemd-journal20:15:000
29623991110,1cyclictest0-21swapper/019:30:140
2962399110,2cyclictest141rcu_preempt23:51:590
2962399110,10cyclictest28797-21ssh23:28:450
2962399110,10cyclictest18191-21rm00:39:360
2963799108,1cyclictest76672sleep320:37:403
2963799108,1cyclictest16373-21cat20:55:213
2963799108,1cyclictest100772sleep320:42:533
2963799107,2cyclictest13543-21uniq20:50:173
2963799106,3cyclictest4994-21expr19:25:133
2963399108,1cyclictest5717-21tr19:25:182
2963399108,1cyclictest26867-21sed20:10:202
2963399108,1cyclictest17677-21taskset21:00:032
2963399108,1cyclictest10853-21ls20:45:132
2963399107,2cyclictest15233-21tr19:45:222
2963399100,1cyclictest141rcu_preempt20:20:162
2962899109,1cyclictest21611-21ls21:05:261
2962899109,1cyclictest14469-21cat20:50:261
2962899108,2cyclictest26367-21cat20:10:161
2962899108,1cyclictest23645-21chrt20:05:051
2962399108,2cyclictest6826-21cat19:30:000
2962399108,2cyclictest28922-21df20:15:170
2962399108,2cyclictest2577-21rm21:55:270
2962399108,1cyclictest31313-21tr20:20:150
2962399108,1cyclictest30779-21cat23:30:010
2962399108,1cyclictest18555-21expr00:20:140
2962399108,1cyclictest11747-21taskset19:40:160
2962399107,2cyclictest3493-21aten2_r7power_v20:30:150
2962399107,2cyclictest13138-21grep20:50:150
2962399100,9cyclictest18417-21rm22:25:000
2962399100,2cyclictest0-21swapper/020:25:140
2962399100,10cyclictest20504-21cat19:55:280
2962399100,10cyclictest20504-21cat19:55:270
296339996,2cyclictest7201-21tr19:30:162
296339996,2cyclictest21026-21sh20:00:002
296339996,2cyclictest21026-21sh20:00:002
296339996,2cyclictest13707-21ls20:50:182
296289997,2cyclictest18943-21aten2_r7power_c19:55:161
296289997,2cyclictest18943-21aten2_r7power_c19:55:161
296239998,1cyclictest22336-21cat21:10:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional