You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-11 - 10:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon May 11, 2026 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
91048321220,5sleep33056-21PK-Backend22:10:223
86419421030,2sleep20-21swapper/220:00:122
139099860,70rtkit-daemon0-21swapper/019:07:290
84515628039,13sleep10-21swapper/119:05:411
84548827340,27sleep30-21swapper/319:09:573
84530026443,15sleep20-21swapper/219:07:312
9608072430,4sleep23056-21PK-Backend00:35:142
8736372280,4sleep10-21swapper/120:25:211
8969562260,3sleep00-21swapper/021:35:010
9339212250,3sleep20-21swapper/223:20:012
9296782240,7sleep2341ksoftirqd/223:05:212
845633992417,5cyclictest883-21dbus-daemon19:22:343
845625992312,9cyclictest945962-21unin-run23:55:001
845633992217,3cyclictest947256-21sed23:55:223
845625992215,6cyclictest883-21dbus-daemon19:58:031
845625992215,6cyclictest883-21dbus-daemon19:58:031
845623992220,1cyclictest3056-21PK-Backend19:15:470
845623992220,1cyclictest0-21swapper/021:40:320
845623992220,1cyclictest0-21swapper/020:36:310
845623992220,1cyclictest0-21swapper/020:26:100
845623992220,1cyclictest0-21swapper/000:02:000
845623992219,2cyclictest0-21swapper/019:25:470
845623992219,2cyclictest0-21swapper/000:30:190
845623992214,6cyclictest933197-21python323:15:210
845623992214,6cyclictest901263-21cut21:45:170
139099220,3rtkit-daemon0-21swapper/023:12:400
84563399210,20cyclictest3056-21PK-Backend20:17:163
84563399210,19cyclictest3056-21PK-Backend21:50:103
84563399210,19cyclictest3056-21PK-Backend21:11:103
84563399210,19cyclictest3056-21PK-Backend20:04:113
84562899213,2cyclictest141rcu_preempt23:50:132
84562899210,20cyclictest0-21swapper/223:46:242
84562899210,20cyclictest0-21swapper/223:46:242
84562899210,20cyclictest0-21swapper/222:48:042
845623992119,1cyclictest0-21swapper/023:33:550
845623992119,1cyclictest0-21swapper/023:21:040
845623992119,1cyclictest0-21swapper/022:42:040
845623992119,1cyclictest0-21swapper/021:51:340
845623992119,1cyclictest0-21swapper/021:03:140
845623992119,1cyclictest0-21swapper/020:12:310
845623992119,1cyclictest0-21swapper/020:05:420
845623992119,1cyclictest0-21swapper/019:56:500
845623992119,1cyclictest0-21swapper/019:56:500
845623992119,1cyclictest0-21swapper/019:51:030
845623992119,1cyclictest0-21swapper/019:21:360
845623992112,8cyclictest865045-21/usr/sbin/munin20:00:200
84562399210,7cyclictest8580662cat19:40:220
84562399210,20cyclictest0-21swapper/023:44:500
84562399210,20cyclictest0-21swapper/022:59:210
84562399210,20cyclictest0-21swapper/021:35:250
139099210,14rtkit-daemon0-21swapper/020:40:590
845633992018,1cyclictest3056-21PK-Backend23:17:303
845633992018,1cyclictest3056-21PK-Backend21:24:143
845633992018,1cyclictest3056-21PK-Backend20:52:393
845633992018,1cyclictest0-21swapper/323:10:413
845633992017,2cyclictest3056-21PK-Backend21:03:393
845633992012,6cyclictest874980-21cat20:30:143
84563399200,1cyclictest3056-21PK-Backend19:41:203
84563399200,19cyclictest0-21swapper/322:02:003
84563399200,19cyclictest0-21swapper/320:35:343
84563399200,19cyclictest0-21swapper/319:12:403
84563399200,18cyclictest3056-21PK-Backend23:21:433
84563399200,18cyclictest3056-21PK-Backend22:38:093
84563399200,18cyclictest3056-21PK-Backend22:07:203
84563399200,18cyclictest3056-21PK-Backend19:55:063
84563399200,18cyclictest3056-21PK-Backend19:55:053
84562899209,9cyclictest883-21dbus-daemon19:22:202
845628992012,6cyclictest902999-21cat21:50:152
84562899200,4cyclictest0-21swapper/221:23:532
84562899200,19cyclictest978-21snmpd22:06:002
84562899200,19cyclictest0-21swapper/223:58:072
84562899200,19cyclictest0-21swapper/223:35:292
84562899200,19cyclictest0-21swapper/223:20:012
84562899200,19cyclictest0-21swapper/222:11:342
84562899200,19cyclictest0-21swapper/222:03:362
84562899200,19cyclictest0-21swapper/221:56:542
84562899200,19cyclictest0-21swapper/220:14:132
84562899200,19cyclictest0-21swapper/219:52:092
84562899200,19cyclictest0-21swapper/219:14:512
84562899200,19cyclictest0-21swapper/200:25:252
845625992017,2cyclictest3056-21PK-Backend23:10:571
845625992013,6cyclictest0-21swapper/119:50:001
845625992013,5cyclictest952936-21sshd00:12:261
845625992013,5cyclictest916461-21cron22:30:001
845625992013,5cyclictest852985-21cron19:30:001
845625992012,6cyclictest880667-21sensors_temp20:45:221
845625992012,6cyclictest436-20systemd-journal20:15:001
84562599200,19cyclictest0-21swapper/121:25:111
84562399203,15cyclictest1095-21runrttasks22:31:110
845623992018,1cyclictest0-21swapper/023:53:560
845623992018,1cyclictest0-21swapper/023:27:530
845623992018,1cyclictest0-21swapper/023:06:240
845623992018,1cyclictest0-21swapper/022:47:170
845623992018,1cyclictest0-21swapper/020:17:250
845623992018,1cyclictest0-21swapper/019:30:200
845623992018,1cyclictest0-21swapper/000:35:470
845623992017,2cyclictest0-21swapper/019:38:040
845623992013,5cyclictest951161-21apt-get00:10:000
845623992013,5cyclictest880978-21sshd20:46:260
84562399200,19cyclictest0-21swapper/022:09:460
84562399200,19cyclictest0-21swapper/022:04:590
84562399200,19cyclictest0-21swapper/020:52:130
84562399200,19cyclictest0-21swapper/020:32:590
84562399200,19cyclictest0-21swapper/020:20:080
84562399200,19cyclictest0-21swapper/000:23:490
845633991917,1cyclictest3056-21PK-Backend23:51:443
845633991917,1cyclictest3056-21PK-Backend23:40:013
845633991917,1cyclictest3056-21PK-Backend23:36:233
845633991917,1cyclictest3056-21PK-Backend23:32:433
845633991917,1cyclictest3056-21PK-Backend23:00:273
845633991917,1cyclictest3056-21PK-Backend21:36:593
845633991917,1cyclictest3056-21PK-Backend21:31:173
845633991917,1cyclictest3056-21PK-Backend20:09:393
845633991917,1cyclictest3056-21PK-Backend00:35:103
845633991913,5cyclictest849140-21seq19:15:223
84563399190,1cyclictest3056-21PK-Backend22:34:113
84563399190,1cyclictest3056-21PK-Backend21:47:133
84563399190,1cyclictest3056-21PK-Backend21:42:063
84563399190,1cyclictest3056-21PK-Backend21:19:213
84563399190,1cyclictest3056-21PK-Backend20:59:543
84563399190,1cyclictest3056-21PK-Backend20:46:093
84563399190,1cyclictest3056-21PK-Backend19:38:563
84563399190,1cyclictest3056-21PK-Backend00:02:253
84563399190,18cyclictest3056-21PK-Backend20:43:593
84563399190,18cyclictest3056-21PK-Backend19:32:003
84563399190,18cyclictest0-21swapper/320:27:573
845628991917,1cyclictest0-21swapper/223:30:152
845628991917,1cyclictest0-21swapper/223:26:512
845628991917,1cyclictest0-21swapper/223:10:352
845628991917,1cyclictest0-21swapper/222:53:182
845628991917,1cyclictest0-21swapper/222:53:182
845628991917,1cyclictest0-21swapper/222:20:192
845628991917,1cyclictest0-21swapper/221:36:122
845628991917,1cyclictest0-21swapper/221:07:462
845628991917,1cyclictest0-21swapper/221:00:162
845628991917,1cyclictest0-21swapper/219:18:162
845628991917,1cyclictest0-21swapper/200:16:192
845628991915,3cyclictest978-21snmpd20:58:432
845628991911,6cyclictest34-21ksoftirqd/200:05:112
84562899190,5cyclictest871063-21sort20:20:012
84562899190,1cyclictest0-21swapper/222:42:502
84562899190,1cyclictest0-21swapper/219:45:332
84562899190,1cyclictest0-21swapper/219:25:532
84562899190,1cyclictest0-21swapper/200:30:042
84562899190,1cyclictest0-21swapper/200:24:232
84562899190,1cyclictest0-21swapper/200:11:392
84562899190,18cyclictest0-21swapper/221:41:192
845625991912,5cyclictest959823-21sshd00:31:361
845625991912,5cyclictest950575-21cat00:05:221
845625991911,6cyclictest903618-21sed21:50:231
845625991910,7cyclictest905207-21python321:55:201
845625991910,6cyclictest857369-21df19:40:141
84562599190,1cyclictest0-21swapper/121:11:121
845623991917,1cyclictest0-21swapper/023:45:510
845623991917,1cyclictest0-21swapper/023:45:510
845623991912,5cyclictest925123-21cron22:55:000
845623991912,5cyclictest925123-21cron22:55:000
845623991912,5cyclictest892596-21needreboot21:20:170
845623991911,6cyclictest1895067-21snapd20:59:130
845623991910,7cyclictest919220-21ntp_states22:35:190
845623991910,7cyclictest889553-21users21:10:200
84562399190,1cyclictest0-21swapper/022:14:320
84562399190,1cyclictest0-21swapper/021:57:290
84562399190,18cyclictest0-21swapper/021:26:470
84562399190,18cyclictest0-21swapper/000:16:330
9243822180,3sleep30-21swapper/322:50:183
9243822180,3sleep30-21swapper/322:50:183
845633991816,1cyclictest3056-21PK-Backend22:57:503
845633991816,1cyclictest3056-21PK-Backend22:15:183
845633991816,1cyclictest3056-21PK-Backend00:06:043
845633991810,6cyclictest1788610-21gdbus00:27:463
84563399180,1cyclictest3056-21PK-Backend23:08:043
84563399180,16cyclictest956231-21users00:20:223
845628991816,1cyclictest0-21swapper/223:44:342
845628991816,1cyclictest0-21swapper/221:16:252
845628991816,1cyclictest0-21swapper/220:48:262
845628991816,1cyclictest0-21swapper/220:27:512
845628991816,1cyclictest0-21swapper/220:06:012
845628991812,5cyclictest902091-21apt-get21:50:002
845628991811,2cyclictest34-21ksoftirqd/222:40:012
845628991811,2cyclictest34-21ksoftirqd/220:22:132
845628991810,6cyclictest875668-21awk20:30:232
84562599185,11cyclictest864559-21df20:00:131
845625991814,3cyclictest846499-21needreboot19:10:181
845625991813,3cyclictest436-20systemd-journal22:15:001
845625991812,5cyclictest957029-21expr00:25:131
845625991812,4cyclictest909064-21sshd22:07:251
845625991812,4cyclictest436-20systemd-journal23:38:261
845625991812,4cyclictest2156-21gnome-shell23:45:091
845625991812,4cyclictest2156-21gnome-shell23:45:091
845625991812,4cyclictest2156-21gnome-shell22:40:091
845625991811,5cyclictest914833-21date22:25:011
845625991811,5cyclictest859723-21cat19:45:211
845625991811,5cyclictest436-20systemd-journal23:33:261
845625991811,5cyclictest1-21systemd21:10:011
845625991810,6cyclictest929043-21grep23:05:141
84562599180,17cyclictest26-21ksoftirqd/121:30:211
845623991816,1cyclictest0-21swapper/023:57:340
845623991816,1cyclictest0-21swapper/000:11:060
845623991812,5cyclictest912251-21sed22:15:210
845623991811,5cyclictest927986-21rt-features23:00:210
845623991811,5cyclictest914481-21apt-get22:25:000
845623991810,6cyclictest860202-21apt-get19:50:000
84562399180,1cyclictest0-21swapper/021:07:000
845633991715,1cyclictest3056-21PK-Backend23:48:053
845633991715,1cyclictest3056-21PK-Backend23:48:053
845633991715,1cyclictest3056-21PK-Backend23:29:203
845633991715,1cyclictest3056-21PK-Backend21:59:023
845633991715,1cyclictest3056-21PK-Backend21:29:083
845633991715,1cyclictest3056-21PK-Backend00:31:323
845628991715,1cyclictest0-21swapper/223:01:572
845628991715,1cyclictest0-21swapper/221:27:332
845628991715,1cyclictest0-21swapper/220:41:572
845628991715,1cyclictest0-21swapper/219:32:022
845628991712,3cyclictest895570-21tr21:30:142
845628991710,5cyclictest856967-21cut19:40:012
84562599179,6cyclictest871496-21df_inode20:20:121
84562599175,10cyclictest436-20systemd-journal21:20:001
845625991715,1cyclictest0-21swapper/119:32:501
845625991715,1cyclictest0-21swapper/100:15:531
845625991713,3cyclictest1095-21runrttasks21:00:041
845625991712,3cyclictest911265-21acpi22:15:111
845625991711,5cyclictest882418-21awk20:50:211
84562599171,14cyclictest960563-21cpuspeed_turbos00:35:121
845625991711,4cyclictest937278-21latency_hist23:30:001
845625991711,4cyclictest901394-21proc_pri21:45:191
845625991710,5cyclictest949400-21cron00:05:001
845625991710,5cyclictest932528-21tr23:15:131
845625991710,5cyclictest926387-21ls22:55:211
845625991710,5cyclictest919198-21ntpq22:35:181
845625991710,5cyclictest883281-21grep20:55:111
845625991710,5cyclictest866899-21cat20:05:221
845623991715,1cyclictest0-21swapper/021:15:050
845623991712,3cyclictest958388-21idleruntime-cro00:30:000
845623991710,5cyclictest939408-21grep23:35:140
845633991614,1cyclictest3056-21PK-Backend00:15:093
845633991614,1cyclictest0-21swapper/300:11:313
845633991612,3cyclictest868405-21ntp_states20:10:183
84563399160,3cyclictest921684-21perf22:45:003
84562899169,5cyclictest436-20systemd-journal23:00:002
845628991614,1cyclictest0-21swapper/222:16:212
845628991614,1cyclictest0-21swapper/220:53:332
845628991611,4cyclictest978-21snmpd00:02:122
845628991610,5cyclictest876617-21expr20:35:132
84562899160,4cyclictest862831-21df19:55:142
84562899160,4cyclictest862831-21df19:55:142
84562599169,5cyclictest947705-21NetworkManager00:00:001
84562599169,5cyclictest934275-21cat23:20:131
84562599169,5cyclictest2176-21JS10:27:481
84562599168,6cyclictest855845-21load19:35:171
845625991613,2cyclictest896889-21latency_hist21:35:011
845625991611,4cyclictest870903-21grep20:20:001
845625991611,3cyclictest1095-21runrttasks23:43:171
845625991610,5cyclictest907689-21sed22:05:001
845625991610,5cyclictest850693-21sed19:20:201
845623991613,2cyclictest3056-21PK-Backend19:10:120
84562399161,11cyclictest915010-21apt22:25:120
84563399153,10cyclictest42-21ksoftirqd/322:25:203
845633991513,1cyclictest3056-21PK-Backend22:47:363
845633991513,1cyclictest3056-21PK-Backend22:20:053
845633991513,1cyclictest3056-21PK-Backend20:23:183
845633991513,1cyclictest3056-21PK-Backend19:54:173
845633991510,4cyclictest859771-21sed19:45:223
84562899159,4cyclictest888751-21expr21:10:112
84562899151,5cyclictest141rcu_preempt19:40:122
845628991512,2cyclictest918215-21sh22:35:002
84562599159,4cyclictest927941-21processes23:00:211
84562599159,4cyclictest876999-21strings20:35:181
84562599158,6cyclictest917114-21df22:30:151
84562599158,5cyclictest892045-21wc21:20:121
84562599152,8cyclictest923677-21apt22:50:121
84562599152,8cyclictest923677-21apt22:50:121
845625991511,3cyclictest26-21ksoftirqd/100:20:171
845625991510,4cyclictest849025-21timerandwakeup19:15:211
845625991510,3cyclictest899363-21kernelversion21:40:161
845625991510,3cyclictest874728-21cpuspeed_turbos20:30:121
84562599150,14cyclictest0-21swapper/120:41:521
845633991412,1cyclictest3056-21PK-Backend21:05:153
845633991410,3cyclictest3056-21PK-Backend19:28:213
845628991410,2cyclictest34-21ksoftirqd/222:30:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional