You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-20 - 04:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Dec 20, 2025 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2778210678,9sleep00-21swapper/019:07:460
270229078,8sleep20-21swapper/219:06:482
282828669,11sleep30-21swapper/319:08:253
281628265,13sleep10-21swapper/119:08:151
304899300,24cyclictest0-21swapper/319:20:113
304499301,20cyclictest225-21systemd-journal21:25:012
304499290,28cyclictest0-21swapper/219:20:112
304499280,27cyclictest0-21swapper/222:45:132
304499280,27cyclictest0-21swapper/222:45:132
304899251,23cyclictest0-21swapper/320:45:123
304499251,20cyclictest225-21systemd-journal19:33:102
304899241,22cyclictest6310-21awk22:40:003
304899241,19cyclictest0-21swapper/321:25:003
304899240,1cyclictest0-21swapper/321:35:153
304499247,12cyclictest0-21swapper/200:15:002
304199240,16cyclictest0-21swapper/119:50:131
304899231,21cyclictest0-21swapper/322:45:133
304899231,21cyclictest0-21swapper/322:45:133
304899230,22cyclictest0-21swapper/323:40:153
304899230,19cyclictest0-21swapper/320:50:123
304899230,19cyclictest0-21swapper/320:50:123
304199220,2cyclictest10580-21/usr/sbin/munin22:45:261
304199220,2cyclictest10580-21/usr/sbin/munin22:45:251
304899210,19cyclictest0-21swapper/322:10:153
304499211,17cyclictest6295-21sendmail22:40:002
304499210,16cyclictest0-21swapper/221:35:142
304899201,17cyclictest11204-21head19:25:173
304499209,8cyclictest21803-21taskset19:47:082
304499202,17cyclictest0-21swapper/220:45:132
304899193,15cyclictest31888-21pmu-power21:15:233
304899192,16cyclictest0-21swapper/319:33:103
304899191,13cyclictest30404-21munin-plugin-st21:15:003
304899190,18cyclictest1777-21dpkg20:15:003
304899190,18cyclictest0-21swapper/320:35:153
304499191,14cyclictest30408-21turbostat.cron21:15:002
304499190,18cyclictest14078-21expr21:45:222
304499190,18cyclictest0-21swapper/222:10:152
304499190,18cyclictest0-21swapper/220:10:132
304499190,13cyclictest11200-21grep19:25:172
304899188,5cyclictest14111-21expr22:55:163
304899181,12cyclictest225-21systemd-journal21:46:523
304499181,16cyclictest0-21swapper/223:40:152
304499180,17cyclictest31887-21tr21:15:232
304199180,17cyclictest0-21swapper/100:20:151
304899177,5cyclictest10636-21sed20:30:263
304899171,14cyclictest0-21swapper/319:40:003
304899170,1cyclictest0-21swapper/320:15:143
304899170,1cyclictest0-21swapper/319:15:153
304899170,16cyclictest0-21swapper/323:30:133
304899170,16cyclictest0-21swapper/321:30:143
304499178,3cyclictest225-21systemd-journal22:42:032
304499176,7cyclictest225-21systemd-journal23:02:042
304499175,7cyclictest31078-21sed22:20:232
304499171,12cyclictest225-21systemd-journal20:28:102
303999175,6cyclictest27255-21munin-run00:30:000
303999170,15cyclictest0-21swapper/021:30:140
304899169,6cyclictest27315-21tr22:15:143
304899161,14cyclictest4956-21unixbench_singl19:10:243
304899160,15cyclictest0-21swapper/320:25:003
304899160,15cyclictest0-21swapper/300:14:593
304899160,12cyclictest0-21swapper/323:10:143
304899160,12cyclictest0-21swapper/322:25:143
304899160,10cyclictest0-21swapper/319:41:273
304499169,6cyclictest6021-21grep20:20:252
304499169,6cyclictest32161-21sed23:30:272
304499169,6cyclictest16384-21wc21:50:222
304499161,13cyclictest537-21snmpd00:20:152
304499160,15cyclictest27786-21tail22:15:172
304499160,15cyclictest0-21swapper/220:35:152
304499160,15cyclictest0-21swapper/200:15:142
304499160,14cyclictest0-21swapper/223:25:222
304499160,14cyclictest0-21swapper/200:00:152
303999164,7cyclictest22018-21awk22:05:000
303999164,6cyclictest2765-21munin-run21:25:010
304899159,5cyclictest8762-21chrt20:27:583
304899159,5cyclictest26984-21head23:20:203
304899159,5cyclictest2579-21chrt23:38:433
304899158,6cyclictest18919-21sort21:55:253
304899153,11cyclictest17437-21apache200:15:153
304899152,12cyclictest21165-21pmu-power19:45:233
304899150,14cyclictest0-21swapper/323:25:223
304899150,14cyclictest0-21swapper/322:05:143
304499158,6cyclictest30285-21tail20:05:182
304499158,6cyclictest24134-21tail23:15:202
304499158,6cyclictest15684-21taskset22:58:402
304499151,13cyclictest2675-21sort20:15:182
304499151,12cyclictest0-21swapper/222:25:162
304499151,10cyclictest4175-21/usr/sbin/munin21:25:212
304499150,14cyclictest7356-21taskset21:33:112
304499150,14cyclictest10438-21/usr/sbin/munin20:30:232
304499150,14cyclictest0-21swapper/223:50:122
304499150,14cyclictest0-21swapper/223:25:002
304499150,14cyclictest0-21swapper/223:10:142
304499150,14cyclictest0-21swapper/200:05:142
304199150,15cyclictest0-21swapper/119:15:231
304899148,5cyclictest30597-21cut20:05:203
304899148,5cyclictest27424-21wc21:05:263
304899148,5cyclictest23955-21sed23:15:183
304899148,5cyclictest16265-21sed20:45:003
304899147,6cyclictest28450-21fgrep20:00:233
304899147,6cyclictest25759-21chrt19:55:213
304899147,6cyclictest12140-21chrt21:40:433
304899141,6cyclictest225-21systemd-journal22:30:163
3048991410,3cyclictest11548-21wc23:55:243
304899140,13cyclictest0-21swapper/300:05:143
304499148,5cyclictest6987-21tr19:15:242
304499148,5cyclictest20780-21sh23:09:592
304499148,5cyclictest11345-21cut22:50:132
304499147,6cyclictest1329-21grep23:35:172
304499141,12cyclictest19423-21gltestperf20:50:162
304499141,12cyclictest19423-21gltestperf20:50:162
304499140,13cyclictest0-21swapper/221:55:142
304499140,13cyclictest0-21swapper/219:35:172
304199146,4cyclictest225-21systemd-journal19:30:011
3041991412,1cyclictest38150irq/131-snd_hda23:30:171
3041991410,3cyclictest2820-21cat21:25:011
304199140,2cyclictest0-21swapper/120:25:141
3039991410,3cyclictest6144-21sed23:45:180
3039991410,3cyclictest2787-21cat22:30:200
303999140,11cyclictest0-21swapper/020:00:140
304899139,4cyclictest39-21ksoftirqd/322:40:133
304899138,4cyclictest291252sleep300:30:273
304899138,4cyclictest27246-21ls00:30:003
304899137,5cyclictest25488-21chrt21:03:083
304899137,5cyclictest22982-21chrt20:57:273
304899137,5cyclictest20534-21taskset23:06:563
304899137,5cyclictest14323-21taskset00:01:303
304899132,9cyclictest20077-21cpu22:00:143
304899132,9cyclictest0-21swapper/323:50:123
304899132,10cyclictest0-21swapper/322:20:123
3048991310,2cyclictest24872-21turbostat.cron00:25:003
304899130,1cyclictest0-21swapper/321:50:203
304499137,5cyclictest26522-21cut21:05:172
304499137,5cyclictest23048-21chrt20:58:172
304499132,7cyclictest225-21systemd-journal00:36:222
304499132,7cyclictest225-21systemd-journal00:36:222
304499131,11cyclictest0-21swapper/222:05:122
3044991310,2cyclictest18517-21wc19:40:182
304499130,13cyclictest0-21swapper/222:00:142
304499130,11cyclictest0-21swapper/220:00:262
304199139,1cyclictest141rcu_preempt20:10:201
304199131,10cyclictest24089-21kworker/1:022:15:131
3041991310,2cyclictest25-21ksoftirqd/123:45:001
3041991310,2cyclictest22888-21grep22:05:151
303999139,3cyclictest29980-21wc00:35:000
303999139,3cyclictest18171-21grep00:10:170
303999133,6cyclictest225-21systemd-journal00:09:550
3039991310,2cyclictest31314-21grep21:15:180
3039991310,2cyclictest31074-21cat20:05:270
304899126,5cyclictest31956-21taskset00:36:583
304899126,5cyclictest31956-21taskset00:36:583
304899126,5cyclictest24399-21cron19:55:003
304899121,7cyclictest225-21systemd-journal23:46:113
304899121,10cyclictest12386-21missed_timers22:50:173
304899120,1cyclictest0-21swapper/321:26:453
304499129,2cyclictest3133-21wc22:30:232
304499129,2cyclictest24479-21wc19:55:002
304499127,4cyclictest25412-21cut19:55:182
304499126,5cyclictest11761-21chrt21:40:222
304499121,9cyclictest0-21swapper/223:45:152
304499120,11cyclictest4955-21awk19:10:242
304499120,11cyclictest0-21swapper/220:40:172
304199129,2cyclictest8382-21sort23:50:161
304199129,2cyclictest4699-21grep20:20:161
304199129,2cyclictest3664-21grep20:15:281
304199129,2cyclictest28016-21cron21:10:001
304199129,2cyclictest24749-21tr19:55:141
304199129,2cyclictest15615-21grep20:40:241
304199129,2cyclictest14454-21grep21:45:261
304199129,2cyclictest12035-21head22:50:161
304199128,3cyclictest38150irq/131-snd_hda00:05:141
304199125,3cyclictest141rcu_preempt23:10:001
304199122,1cyclictest0-21swapper/100:15:141
3041991210,1cyclictest8127-21tr21:35:151
303999129,2cyclictest9266-21grep19:20:190
303999129,2cyclictest7224-21grep22:40:170
303999129,2cyclictest4906-21grep22:35:180
303999129,2cyclictest4591-21wc19:10:200
303999129,2cyclictest32221-21systemctl00:40:000
303999129,2cyclictest32221-21systemctl00:39:590
303999129,2cyclictest30735-21tr23:30:140
303999129,2cyclictest23962-21sed00:20:230
303999129,2cyclictest22471-21cron00:20:000
303999129,2cyclictest13513-21head21:45:170
303999128,3cyclictest5076-21grep20:20:170
303999128,3cyclictest29213-21cat23:25:200
303999128,3cyclictest26421-21sed19:55:270
303999128,3cyclictest12456-21grep22:50:190
303999128,2cyclictest5095-21turbostat.cron23:45:000
304899112,6cyclictest225-21systemd-journal23:00:163
304499115,5cyclictest26687-21sed00:25:262
304499110,7cyclictest1142-21runrttasks00:30:062
304499110,10cyclictest24578-21needreboot21:00:202
304199119,1cyclictest10594-21grep23:55:141
304199118,3cyclictest16013-21cat21:50:181
304199118,2cyclictest6841-21grep22:40:151
304199118,2cyclictest29970-21grep23:25:271
304199118,2cyclictest2891-21grep22:30:211
304199118,2cyclictest26528-21grep23:20:161
304199118,2cyclictest21866-21head20:55:191
304199118,2cyclictest20662-21grep19:45:181
304199118,2cyclictest19063-21cat00:10:261
304199118,2cyclictest17560-21apache_volume19:40:121
304199118,2cyclictest17283-21ls23:00:221
304199118,2cyclictest13096-21cat20:35:271
304199118,2cyclictest11307-21dump-pmu-power20:35:001
304199117,3cyclictest9102-21ls19:20:171
304199117,3cyclictest6088-21head23:45:171
304199115,5cyclictest11133-21grep21:40:161
304199112,1cyclictest141rcu_preempt00:00:171
304199111,7cyclictest38150irq/131-snd_hda22:25:141
304199110,10cyclictest30334-21tr00:35:151
304199110,10cyclictest30334-21tr00:35:141
303999119,1cyclictest32637-21head20:10:170
303999119,1cyclictest30495-21cat22:20:170
303999119,1cyclictest24201-21grep23:15:200
303999118,2cyclictest9702-21grep22:45:170
303999118,2cyclictest9702-21grep22:45:170
303999118,2cyclictest799-21grep23:35:150
303999118,2cyclictest7860-21cat20:25:190
303999118,2cyclictest4893-21taskset21:27:040
303999118,2cyclictest21569-21sed19:45:280
303999118,2cyclictest16885-21sed20:45:150
303999118,2cyclictest16203-21basename23:00:150
303999118,2cyclictest15893-21ls23:00:010
303999118,2cyclictest15792-21chrt21:50:170
303999118,2cyclictest1530-21sh22:30:000
303999118,2cyclictest13520-21grep19:30:170
303999118,2cyclictest12641-21tr00:00:150
303999115,2cyclictest225-21systemd-journal20:40:000
303999113,4cyclictest225-21systemd-journal21:10:010
303999110,1cyclictest0-21swapper/023:05:130
304499100,9cyclictest11193-21tr23:55:182
304199108,1cyclictest881-21cat23:35:151
304199108,1cyclictest6175-21grep21:30:181
304199108,1cyclictest31643-21cat21:15:201
304199108,1cyclictest30046-21diskstats22:20:131
304199108,1cyclictest28660-21expr21:10:141
304199108,1cyclictest27795-21cut00:30:171
304199108,1cyclictest27463-21expr20:00:151
304199108,1cyclictest25293-21chrt23:17:051
304199108,1cyclictest21945-21taskset22:04:271
304199108,1cyclictest18413-21taskset20:49:451
304199108,1cyclictest17009-21chrt19:36:481
304199107,2cyclictest30265-21head20:05:181
304199107,2cyclictest26922-21taskset00:26:031
304199107,2cyclictest21192-21tr23:10:161
304199107,2cyclictest14875-21seq19:34:541
304199102,1cyclictest141rcu_preempt20:55:001
304199102,1cyclictest141rcu_preempt20:55:001
303999108,1cyclictest7942-21taskset21:35:030
303999108,1cyclictest6466-21ls19:15:190
303999108,1cyclictest27873-21taskset21:08:140
303999108,1cyclictest27770-21seq23:22:210
303999108,1cyclictest22675-21cat19:50:140
303999108,1cyclictest21827-21grep23:10:180
303999108,1cyclictest21562-21expr20:55:170
303999108,1cyclictest18886-21sensors21:55:230
303999108,1cyclictest17537-21acpi19:40:110
303999108,1cyclictest11772-21grep20:35:150
303999108,1cyclictest10515-21wget21:40:130
303999107,2cyclictest9918-21grep20:30:180
303999107,2cyclictest7986-21tr23:50:140
303999107,2cyclictest24978-21expr22:10:160
303999107,2cyclictest24399-21wc22:09:590
303999107,2cyclictest23964-21df_abs21:00:160
303999107,2cyclictest2280-21cut20:15:160
303999107,2cyclictest15277-21aten2_r7power_v19:35:170
303999103,4cyclictest27077-21apt-get22:15:140
303999100,9cyclictest204552sleep020:50:270
303999100,9cyclictest204552sleep020:50:270
30419998,1cyclictest18736-21awk21:55:211
30419997,1cyclictest4726-21cat19:10:231
30419997,1cyclictest4492-21cat22:35:161
30419997,1cyclictest256282sleep121:04:541
30419997,1cyclictest25238-21fschecks_count22:10:181
30419997,1cyclictest15488-21chrt22:56:091
30419991,6cyclictest0-21swapper/121:25:141
30399997,1cyclictest12174-21taskset19:25:400
30399996,2cyclictest11037-21uname23:55:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional