You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 04:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Nov 27, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1120221320,0sleep10-21swapper/121:28:231
2735429668,9sleep00-21swapper/019:08:510
2723329581,9sleep10-21swapper/119:07:151
2715329582,9sleep30-21swapper/319:06:123
2738828665,6sleep20-21swapper/219:09:172
199812540,1sleep10-21swapper/123:48:001
2753499282,25cyclictest16597-21diskmemload21:39:101
2753499280,27cyclictest10871-21turbostat00:05:001
2753499280,27cyclictest0-21swapper/123:38:591
27534992717,9cyclictest12495-21nfs4_client00:05:201
2753499270,25cyclictest0-21swapper/121:40:111
2753499260,24cyclictest0-21swapper/100:36:441
2753499260,1cyclictest0-21swapper/122:54:351
2753499250,24cyclictest0-21swapper/122:14:481
2753499250,23cyclictest0-21swapper/123:41:041
2753499250,1cyclictest0-21swapper/122:35:141
2753499243,14cyclictest26796-21ssh23:30:171
2753499242,21cyclictest16597-21diskmemload23:53:381
2753499241,17cyclictest31315-21aten2_r7power_c21:20:131
2753499240,23cyclictest0-21swapper/122:05:191
2753499240,1cyclictest0-21swapper/100:10:131
2753999230,23cyclictest0-21swapper/323:01:023
2753499233,19cyclictest330-21ssh00:20:231
2753499230,22cyclictest0-21swapper/121:54:371
2753499230,22cyclictest0-21swapper/121:30:101
2753499230,20cyclictest24158-21cpuspeed_turbos21:15:131
2753499230,20cyclictest0-21swapper/123:15:531
2753499230,20cyclictest0-21swapper/122:40:031
2753499230,1cyclictest0-21swapper/123:23:111
2753499230,1cyclictest0-21swapper/122:49:481
2753699220,21cyclictest0-21swapper/222:01:012
2753499221,1cyclictest0-21swapper/121:10:131
2753499220,21cyclictest0-21swapper/122:02:081
2753499220,21cyclictest0-21swapper/100:31:081
2752999222,19cyclictest8059-21ssh23:40:110
27536992118,2cyclictest0-21swapper/221:58:002
2753499213,17cyclictest17972-21sh23:01:351
2753499212,17cyclictest16597-21diskmemload22:57:131
2753499211,19cyclictest31053-21pmu-power23:55:211
2753499211,16cyclictest0-21swapper/119:55:191
2753999200,19cyclictest0-21swapper/321:15:563
27536992020,0cyclictest0-21swapper/221:52:032
2753699200,2cyclictest0-21swapper/223:41:562
2753699200,19cyclictest0-21swapper/222:28:562
2753499204,15cyclictest16597-21diskmemload23:06:321
2753499202,17cyclictest20612-21unixbench_multi23:25:291
2753499200,19cyclictest0-21swapper/121:56:111
2753499200,17cyclictest1842-21grep19:20:281
2753699191,17cyclictest0-21swapper/220:34:552
2753499191,1cyclictest0-21swapper/100:18:361
2753499191,15cyclictest16597-21diskmemload22:15:571
2753499190,1cyclictest0-21swapper/119:25:131
2753499190,18cyclictest0-21swapper/100:28:151
2753999180,13cyclictest27651-21ssh21:40:003
27536991815,1cyclictest0-21swapper/200:15:032
2753499181,15cyclictest31529-21grep20:30:011
2753499180,1cyclictest16597-21diskmemload22:30:231
2753499180,1cyclictest0-21swapper/120:40:101
2753499180,1cyclictest0-21swapper/119:45:131
2753499180,18cyclictest0-21swapper/121:45:361
2753499180,14cyclictest16597-21diskmemload22:25:261
2753699170,2cyclictest0-21swapper/221:35:252
2753699170,1cyclictest141rcu_preempt22:23:432
2753699170,1cyclictest0-21swapper/223:37:182
2753699170,14cyclictest0-21swapper/222:54:152
2753499179,6cyclictest29030-21tr20:20:131
2753499173,11cyclictest31763-21apt-get20:30:081
2753499172,1cyclictest16597-21diskmemload23:10:291
2752999176,6cyclictest491-21latency_hist23:35:000
27536991615,1cyclictest0-21swapper/223:29:082
2753699160,1cyclictest0-21swapper/221:10:162
2753699160,15cyclictest0-21swapper/223:23:282
2753699160,13cyclictest0-21swapper/221:46:292
2753699160,13cyclictest0-21swapper/200:08:512
2753499161,12cyclictest27829-21runrttasks22:24:161
2752999160,14cyclictest0-21swapper/021:26:000
2753999150,7cyclictest20056-21ntp_states23:25:243
27536991512,2cyclictest22780-21tr20:05:222
2753699150,15cyclictest0-21swapper/223:56:262
2753699150,15cyclictest0-21swapper/222:11:582
2753699150,12cyclictest0-21swapper/223:33:462
2753499159,5cyclictest13084-21wc21:00:191
2753499158,6cyclictest9564-21date20:55:001
2753499152,11cyclictest9721-21cut20:55:131
2753499152,11cyclictest0-21swapper/120:05:121
2753499152,10cyclictest21272-21cron20:05:001
2753499150,1cyclictest0-21swapper/121:05:191
2753499150,1cyclictest0-21swapper/119:40:131
2753499150,14cyclictest0-21swapper/119:10:131
2752999151,3cyclictest0-21swapper/021:05:130
27529991513,2cyclictest0-21swapper/023:38:320
2753699141,3cyclictest5871-21latency_hist21:25:002
27536991410,3cyclictest28937-21sed21:40:162
27536991410,2cyclictest14355-21cat21:05:002
2753699140,3cyclictest0-21swapper/200:21:132
2753699140,14cyclictest37850irq/131-snd_hda22:56:072
2753699140,14cyclictest0-21swapper/222:35:512
2753699140,13cyclictest0-21swapper/221:34:002
2753499148,5cyclictest8816-21wc19:35:261
2753499148,5cyclictest6255-21grep20:45:221
2753499148,5cyclictest27735-21readlink20:15:241
2753499148,5cyclictest1968-21taskset20:36:051
2753499140,1cyclictest0-21swapper/120:10:151
2752999149,5cyclictest20902-21ssh21:56:400
2752999146,7cyclictest7231-21cron22:55:000
2752999144,9cyclictest5737-21python23:15:240
2752999140,2cyclictest0-21swapper/021:36:250
2752999140,13cyclictest4179-21grep20:40:210
27539991310,2cyclictest21581-21cut20:05:143
27539991310,2cyclictest14566-21wc23:00:003
2753699139,3cyclictest6495-21grep19:30:262
2753699139,2cyclictest22055-21rm23:50:012
2753699131,2cyclictest0-21swapper/219:45:132
27536991310,2cyclictest6377-21tr21:25:142
27536991310,2cyclictest5967-21grep23:15:262
27536991310,2cyclictest5178-21grep00:00:182
27536991310,2cyclictest3199-21tr19:25:162
27536991310,2cyclictest27942-21grep22:45:272
2753699130,13cyclictest0-21swapper/223:50:082
2753499132,7cyclictest224-21systemd-journal19:55:001
2753499130,4cyclictest224-21systemd-journal19:17:001
2753499130,12cyclictest0-21swapper/119:30:121
2752999139,3cyclictest3101-21grep21:45:140
2752999139,3cyclictest14202-21cron21:05:000
2752999138,1cyclictest0-21swapper/000:25:150
2752999133,5cyclictest13992-21perf22:15:000
2752999131,6cyclictest23481-21munin-run21:15:000
27529991310,2cyclictest27832-21grep19:10:140
27529991310,2cyclictest26214-21grep22:00:210
27529991310,2cyclictest19356-21sh23:25:180
27529991310,2cyclictest17947-21grep22:40:000
2752999130,12cyclictest0-21swapper/000:00:140
2753999129,2cyclictest827-21ssh22:50:153
2753999129,2cyclictest8037-21ssh23:40:113
2753999129,2cyclictest7548-21grep20:50:143
2753999129,2cyclictest6871-21cat00:25:303
2753999129,2cyclictest6565-21grep19:30:263
2753999129,2cyclictest4693-21cat21:45:273
2753999129,2cyclictest4628-21grep22:30:203
2753999129,2cyclictest4135-21cat20:40:213
2753999129,2cyclictest396-21sed22:05:163
2753999129,2cyclictest39-21ksoftirqd/323:49:113
2753999129,2cyclictest30428-21grep20:20:243
2753999129,2cyclictest29370-21grep22:25:153
2753999129,2cyclictest28021-21grep20:15:283
2753999129,2cyclictest27177-21ssh22:45:213
2753999129,2cyclictest26929-21grep23:30:183
2753999129,2cyclictest23437-21wc23:05:213
2753999129,2cyclictest19929-21grep20:00:193
2753999129,2cyclictest11767-21awk19:45:003
2753999129,2cyclictest11757-21tr21:50:213
2753999129,2cyclictest11455-21tr22:35:163
2753999128,3cyclictest22107-21wc00:40:003
2753999127,4cyclictest39-21ksoftirqd/320:50:003
2753999125,3cyclictest224-21systemd-journal21:10:003
27539991211,1cyclictest0-21swapper/322:19:003
27539991211,1cyclictest0-21swapper/321:20:123
2753699129,2cyclictest9452-21basename20:55:002
2753699129,2cyclictest3624-21wc20:40:152
2753699129,2cyclictest28507-21sleep20:20:002
2753699129,2cyclictest25680-21head20:10:272
2753699129,2cyclictest19881-21grep20:00:192
2753699129,2cyclictest19149-21sort22:40:152
2753699129,2cyclictest15773-21grep22:15:202
2753699129,2cyclictest14759-21cut21:05:142
2753699129,2cyclictest10802-21cut20:55:192
2753699128,3cyclictest700-21sh22:05:182
2753699128,3cyclictest2290-21ls20:40:002
2753699123,5cyclictest224-21systemd-journal19:55:002
2753699123,4cyclictest224-21systemd-journal22:30:002
2752999129,2cyclictest7067-21apt20:50:000
2752999129,2cyclictest32691-21grep00:20:210
2752999129,2cyclictest30113-21grep20:20:210
2752999129,2cyclictest27917-21head22:45:270
2752999129,2cyclictest27081-21grep20:15:180
2752999129,2cyclictest26500-21grep00:15:270
2752999129,2cyclictest25308-21ls21:15:200
2752999129,2cyclictest22220-21grep20:05:170
2752999129,2cyclictest18922-21head00:10:170
2752999129,2cyclictest17539-21grep00:35:200
2752999129,2cyclictest17317-21df19:55:170
2752999129,2cyclictest1713-21sed22:05:260
2752999129,2cyclictest15097-21grep19:50:170
2752999129,2cyclictest13231-21sed23:20:250
2752999129,2cyclictest12511-21grep00:30:270
2752999129,2cyclictest12320-21ssh21:50:260
2752999128,3cyclictest24546-21grep20:10:180
2752999127,2cyclictest493-21systemd-logind00:08:570
2753999119,1cyclictest30513-21grep23:10:193
2753999118,2cyclictest6654-21rm00:00:443
2753999118,2cyclictest6383-21tr21:25:143
2753999118,2cyclictest5020-21taskset23:37:333
2753999118,2cyclictest4066-21ssh23:15:083
2753999118,2cyclictest3581-21ssh23:59:433
2753999118,2cyclictest31807-21cut20:30:093
2753999118,2cyclictest30008-21sed21:40:263
2753999118,2cyclictest29119-21ssh23:54:543
2753999118,2cyclictest27300-21ssh22:00:473
2753999118,2cyclictest24695-21expr20:10:183
2753999118,2cyclictest21547-21rm22:20:033
2753999118,2cyclictest21537-21taskset21:57:153
2753999118,2cyclictest18929-21head00:10:173
2753999118,2cyclictest18921-21lockfile-create20:00:003
2753999118,2cyclictest18533-21tr22:40:133
2753999118,2cyclictest17470-21expr21:10:153
2753999118,2cyclictest16497-21cron19:55:003
2753999118,2cyclictest16329-21sh23:23:143
2753999118,2cyclictest14109-21sh00:32:153
2753999118,2cyclictest13864-21chrt19:46:433
2753999118,2cyclictest13497-21rm22:14:353
2753999118,2cyclictest11306-21grep20:55:253
2753999114,6cyclictest3261-21kworker/u8:021:30:193
2753699118,2cyclictest29960-21grep20:20:202
2753699118,2cyclictest18879-21sh20:00:002
2753699117,2cyclictest10970-21cut00:30:142
2753699116,4cyclictest5473-21aten2_r7power_v00:25:172
2753699114,3cyclictest224-21systemd-journal21:15:002
2753699110,1cyclictest0-21swapper/200:37:552
2753699110,10cyclictest21256-21ssh23:04:422
2753699110,0cyclictest0-21swapper/223:07:442
2752999118,2cyclictest7976-21grep19:35:180
2752999118,2cyclictest6776-21chrt19:32:300
2752999118,2cyclictest4088-21sort00:00:000
2752999118,2cyclictest32394-21ssh22:26:420
2752999118,2cyclictest32334-21grep21:20:180
2752999118,2cyclictest31831-21ssh23:10:580
2752999118,2cyclictest31489-21cron20:30:000
2752999118,2cyclictest28854-21grep21:40:160
2752999118,2cyclictest26822-21sh23:52:400
2752999118,2cyclictest2210-21taskset20:39:100
2752999118,2cyclictest19597-21grep22:40:180
2752999118,2cyclictest16705-21ssh21:31:430
2752999118,2cyclictest15195-21grep22:15:160
2752999118,2cyclictest13128-21ssh22:58:470
2752999118,2cyclictest12782-21sed19:45:160
2752999110,3cyclictest0-21swapper/023:47:330
167862110,0chrt0-21swapper/300:09:143
2753999108,1cyclictest2946-21cat19:25:143
2753999108,1cyclictest28359-21fschecks_count19:10:163
2753999108,1cyclictest19162sleep320:35:243
2753999108,1cyclictest14148-21seq21:04:223
2753999107,2cyclictest853-21ssh00:20:273
2753999107,2cyclictest31122-21seq20:25:193
2753999107,2cyclictest24586-21ssh00:15:073
2753999107,2cyclictest1979-21taskset19:21:533
2753999101,3cyclictest224-21systemd-journal19:36:113
2753699108,1cyclictest5515-21sed20:45:162
2753699108,1cyclictest310982sleep220:25:012
2753699108,1cyclictest29657-21cat23:10:142
2753699108,1cyclictest207382sleep200:11:132
2753699108,1cyclictest1706-21sed19:20:282
2753699107,2cyclictest7955-21head19:35:182
2753699107,2cyclictest30112-21acpi19:15:112
2753699107,2cyclictest28477-21tail19:10:172
2752999108,1cyclictest32103-21expr20:30:100
2752999108,1cyclictest21750-21sh22:20:130
2752999108,1cyclictest1318-21cut19:20:230
2752999108,1cyclictest10356-21expr19:40:160
2752999107,2cyclictest8104-21sed20:50:160
2752999107,2cyclictest3672-21ssh22:30:070
2752999107,2cyclictest30971-21basename19:15:170
2752999107,2cyclictest2783-21grep19:25:140
2752999107,2cyclictest21872-21ssh23:05:040
2752999107,2cyclictest19122-21seq20:00:030
2752999107,2cyclictest16768-21grep23:00:260
2752999107,2cyclictest11899-21grep21:00:000
275399997,1cyclictest30734-21cat19:15:153
275369997,1cyclictest9766-21basename19:40:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional